IDEA Technical Report No. 17 


Technical Manual for the 
Revised IDEA Feedback for 
Administrators System 


Stephen L. Benton 
Amy B. Gross 
William H. Pallett 
Jihyun Song 
Russell Webster 
Meixi Guo 


December, 2011 


Table of Contents 


INTRODUCTION sicec2hscieccesceetesseds ic; cee S2b Ase cE cb taedbe ke labic reece dees ih cced cu deeesabcoiccdcdepdeeeedbsedsacateudesdideselcaeese 1 
HISTORICAL: BACKGROUND i. sieciiiccccesetecccceses ccc csotedcdedeteces coeused shed dekewautecdeleleadetwedesadecselceschwedesddeciecdcaceds 2 
REVISION OF THE IDEA FEEDBACK FOR ADMINSTRATORS SYSTEM...........ccccccsssssssssssceeeccccssssssssceeseeeeenes 3 
Analysis of the 2001-2008 Database............cccccscccccssssssssececeeecsssesseaeeeecesseeaaeeeesesceseaaeaeeeeseessesaaaesesesssesaaeas 3 
Demographic Information — Administrative Units and INStitutiOns: .......c.cccccccccccsesssssceseceeessssssaeeeseesees 3 
Demographic Information — Administrators and their Units: ....cccccccccccccsssssscccesessscssaecesecessssnsseeeesensees 4 
Demographic Information — RESPONCENES: ......ccccccceessssccecesseessnesecececeseesnaeeeeeceseuseaaeaeeeessusesusaeeeeeesseesaas 5 


Correlations between Number of Personnel Asked to Respond, Response Rates, and Ratings of the 


AGINIMISE LONE sescccicadeviecesdceuteaictelvaced eee isles Beea deca Sieh ce Poza toe abate beanie anaes Gad agate aee ee eG 6 
Administrator Perceptions Of the Unit: ..c.cccccccccccsssssccccescesssnsceeececessesnseeeeecessesesaeseeesecusesasaeceesesseessaaeaeess 9 
SUITING Y os easses et eben shades codde oc daa cade La0k gece aTekdasEaa ye awe dalees ea decd sc osabbas 025040 aS yea vaT dade s au Dads evdach doweid. daives 11 
Analysis of 2001-2008 Database: Descriptive Statistics .............cccccscccccesssssssececeeecessessseeeeeeessessaeeeeeeseesees 11 
Personnel Ratings of the Administrator’s Performance of Administrative ROl@S: ........ccccccccceesssceeseees 11 
Ratings of the Administrator’s Personal ChHAracteristiCs: ......cccccccccccccsccseescescsecececcecuceseeuecsescsesseeseneees 13 
Summary Judgments of the AAMINiStratOr: ......cccccccccccccsessscecececeseesecececececuscaaaeeeeeesseesnaeaeeessussesnsaeees 14 
SY] 41) £019] AAPOR CEEPESPRTCE EO CPE TT CRT CTE CECE CEE OTTER OEE RTE CECE CREEPER CCU Or TEC OTTO CERT TTT COREE CREEP PR OSTCET 16 
User'dnd Expert:REVICW iesiesccielaeiis eliese 8 cos cSeeecahesa ds res Beh case deel UW lees ah em eae see hs eS 16 
Inptit: from FOCUS: GROUP s..oisscogece S555 6ss sack cs ec Sed be ca ae chac ai ca toss auce be Gis ba Un sBaib bo 43s Bas doth ds N44 sabi Ba Lhbs be bao TS 16 
Input from-Expert PANel cvs: svecccicccsccsesvsaiecsvaccessovanasnnsvessussedensdeaiebedeateaysessdtvasdoasnessateseranbecTeassesvebadenses 17 
i =1qq i BY=N'(=)10] ©] a0 =] 0] Renee ener eee rn ee eee 18 
Th@AGMINIStrAtOF S)ROIOS? i. cesscis eiecacceceesusscesacaseze cide suadeaavecesiWaieeNansseestWabete teas vbesedasyetetadscteiWarsbicdaasteds 18 
The Administrator’s Leadership Style and Personal Characteristics: ........cccccceesesscceseeseesssceceeeeeeesesaes 19 
Overall Impressions Of the ACMINIStIAtOM: ........cccccsccccccesesesseceeeceecsseaaeeesessseseaaeseeeeesesseaaeeeeseuseessaaeess 21 
OPeN-CNded COMIMENAS: ......ccccccccccccecssssnsceceeeeeseesenneeeeeeeeseeaeaeeeeeeeseeeeaeeeeeeceseesaaaeseeesssuseaaeaeeessessenaaaaas 22 
Administrator INfOrmation FOr: .....ccccccecsccccccecesessssecesecssccsesaeeeeessesseeaeeeeesesessaueeeeeessessusaaaeseesssseaaaeas 23 
Process: MOIfI CATION: steccis sit evceseectceseesstieasaecs cesgenscesetes ciesueneteesdeus seb¥eus taeda ath cvsueutaasysatsdedhanetersbeus seats dees 25 
Optional Gap-An lysis: 5 elecivesicedasszcesasesecesesesces eta chainces nsec beee see oa Paw AEG AE ca RE og 25 
Optional SUD-GroUP;ANGIVSISS <csrccccccesia seeds isspesbeaki cde v bosses eveths vos bobcat scab ds Gol des end cde vdsadea ease veevss eee 25 
REDOPE. EMI PNGSIS 22sec Ses deec a iceecs Aiea Nee aS abe as 26 
VALIDITY eecscctivccccceiesevcciccckcheesscstesdictcscesssssscatiticctooss'sssssiicccdCessssssacsisetcstebesssseciiscccboesssssesdiescecensssseatinse 26 
Evidence of Content Validity: sc.ccsissescessascsseavedestes sacs ves saveceandicewsatva scneeds yeenendsdasdadeoeeedsceseisaaseoeseaSadeleeacasesseas 26 
Validity Evidence Based on Internal Structure ...........cccecsssccccessssssececececessesssaeececesecseaeaeeeeeesseseaeeeseesseeeees 27 
AQMINISHrALIVE ROLES reivcscicccvessccec ves euveacdacuveascdcnvescden vei cde civeadcceuveaddcuveacdd cuveudedcavencaccucacascivestecedveacadeveacs 27 
P@FSONGL: CNALACCELISTICS Hs scccivceseeeiads ceed cistets tees selahetablee debdee esac’ Madedets cated ease ablaeeatdeeeeeari eat ees 30 
SUITING Y Sac svertacce cbc acetaved set cth chit cevssenasctastecsteavence cedanatustavavert shah vandgves stuatedsteaveasecesacneathaieeateriat ten oteitedy 34 
Validity Evidence Based on Relationships to Other Variables ............cccccccssssccccecssseseseeeeeeesesscseseeeesensees 35 
Comparison of Factor Composite Scale Scores by Demographics of the Administrator and 
Perceptions Of: the UNibs «chides Aockak os tees teed echt cleats eta eA PR ORE a Rn 35 


i 


Comparisons of Summary Judgments by Demographics of the Administrator and the 
Administrator’s Perceptions Of the Unit: .......ccccccssssccccecssesssssceceeecessncuaaeeeecessesnaaeeeeeessessaaeaeeeeseessesnaaeas 36 


Comparisons of Personnel Ratings of Administrative roles and Personal Characteristics by Rater 


Comparisons of Summary Judgments by Rater Perspectives: ........cccsesssccccceessensececeeecessessaeaeeceseesnnaeas 41 


Differences in Ratings of Overall Impressions between Raters Responding to, and those omitting 


Responses to, Rater Perspective QUESTIONS: ....cccccccccceceeeceseseseeeseeesesesesesesuseeeneaeusasasasasusasasasaeaeaeaeaeanaes 44 
Additional Investigation of Rater Perspective ItCMs: ........ccccccccccecesessseceseeesessncuaueeescessscusaeeeeeessesseaaaes 44 
SITING aos Sarasin ye 05 lee ta aes Caen Das en ek AG Ok as a OS I Ea ala PRR a STE a BERS wah i os CE Reed gdh ta aaa ca ch See aha 45 
RELIABILITY AND STANDARD ERRORS OF MEASUREMENT .............sssssscccccsssssssssesceeccssessssssssceeeeesessess 46 
Estimates of Item Split-Half Reliabilities and Standard Errors of Measurement on Ratings of 
Administrative: Roles isccvesccccccceesccodsavciedessescsacestisdscesessanssevedsdeseudsasvesvdeccuetebssaatesvccceuscesasssteuedecesessaesy 46 
Estimates of Item Split-Half Reliabilities and Standard Errors of Measurement on Ratings of Personal 
ClaAPacteriStiCsy.:: rcucsiieecesteecnacdes tensscedawessce cenasthsnanesessyaeesaesunacteslaaduets laud sbasincsdesvaasdediiaaveaestaadeanueevadwasscenees 47 
Estimates of Item Split-Half Reliabilities and Standard Errors of Measurement on Overall Summary 
JUABIMENTS ies. cc.ccsteceecesscecadsaecsslatuaca cesucheacsnana sicanticacesuantessuaescausbiace casvabeestiaadeasattaccedeihiseshoneaensecetuaoe anaes 48 
Internal Consistency Of Scale ............ccccccccssssecesseneeceeseacececeaaeceeseaaececeeaaecesesaeeeseeaaececeeaaeceseeaeeeseeaaeeseeeaas 49 
Estimates.of Item Stability ./..2.cccccclilececsecacecevecsalessicccastecncasdecesaeeduia cde evaaseceveasasessit chet scvceesueasiaatuinevaanvere 50 
SUM y os eses. eassve. Se ieceescsaccis latest csc eaca tessacias Sacavecal oes i vials aaa Ae ies oad she dade, 51 
REFERENCES ‘esccsesscdccssccscestesocticcetacess oesvecsikenscacecsbssecsseccscacuebied cisccsaaedbebedscesasestacdceustscecesscade sesvesscnscecaces 52 
APPENDIX A: ADMINISTRATOR INFORMATION FORM AND IMPRESSIONS OF ADMINISTRATOR ........ 53 
APPENDIX B: REVISED ADMINISTRATOR INFORMATION FORM AND IMPRESSIONS OF ADMINISTRATOR 
abaldssibcccBitetsleishscecesdasieldaladecwedddbatasstsetoendeseeshslace casaddbbslesscdse covaddedsbessescucceddushelscaancteStuetendtaseousasdabelasanncs 54 
APPENDIX'C: FOCUS GROUP THEMES. ..ecccsscscccccccssdissesesecaceceticssssscsecacesseaubessssessccsedesesessceseccesedeseserceees 55 
APPENDIX'D: EASE 2isciciccscc cc edsdelchscatecu cece esesdedic ceasbebetsistacdsc2ab cheisclan catesdieielsclanseveddetebendsaeucesdessiestance 58 
APPENDIX E: SAMPLE REVISED REPORT..........cccccssssssssceccccccssssssssceeeccecasssssssceesccecacaasssscesesseeasaaassssseees 59 


List of Tables and Figures 


Table 1: Frequencies and Percentages of Institutions Representing Various Accreditation Regions......... 3 
Table 2: Means and Standard Deviations for Unit Size and Personnel Response Rate..........sssssssccceseseenee 4 
Table 3: Frequencies and Percentages of Participating Administrators’ Institutional Enrollments and 
TUDTCION tacit ahaa tata aa tenutheceeb heat Seeded A deetin Seas esto nda tee ee ae aaaaaledl 4 

Table 4: Frequencies and Percentages of Ranges in Administrators’ Years Of SCrviCe .....s:cccccccccccsessssseeeees 5 
Table 5: Frequencies and Percentages of Numbers of Support Staff and Professionals in the Unit........... 5 
Table 6: Frequencies and Percentages of Various Personnel! CHAracteristics .......cccccccccecssstsececessessstsceeeeeens 6 
Table 7: Correlations between Respondent Ratings, Number Asked to Respond, and Response 

ROC ic csvecesceecevnndeesencbese see saves ne dave sand eau sues exes cua eee daub iu evave dieu tudes Diatateea vaveee san edgieenbee rand eeaeeeens 8 
Table 8: Frequencies and Percentages of the Administrator’s Perceptions of the Unit’s 

Characteristics 10 
Table 9: Percentage of Administrators Falling Within Mean Category Ratings of 10 

AGIMINIStFatiVe: ROLES. scccinesadeceevscce dees soceved ive ut'al dca otdeeuteoudiagtevadecevsoteceutaaehegueduticevasdeciesbiesveisiecties 12 
Table 10: Means and Standard Deviations for A-Priori Administrative Role Subscales and Items............ 13 
Table 11: Means and Standard Deviations for A-Priori Administrative Style Subscales and Items........... 14 
Table 12: Percentages of Administrators Falling Within Ranges of Category Ratings on Item 29............ 15 
Table 13: Percentages of Administrators Falling Within Ranges of Category Ratings on Item 30............ 15 
Table 14: Progression of Wording of Administrative roles Retained in the IA-R ......ccccccccssssesssssccesesststseees 19 
Table 15: Administrative Style and Personal Characteristics DIMENSIONS .......csssssssscccecessssnsssesecesesntssaes 20 
Table 16: Progression of Wording of Leadership Style and Personal Characteristics Items in the IA-R_ ...21 
Table 17: Progression in Wording of Overall IMpression ItCMS ........ccscsessssecccensnsssscececesssnssssssecscssnsnsaasasees 22 
Table 18: Progression of Wording in Open-ended COMIMENEHS .u.cssccccccccsssssesssececsseseesscesecscssestsaeeesscsseseeaaes 22 
Table 19: Progression of Wording on Research Items included on the Administrator Information 

POUIWE icv scdevuistedeenes sig oan sdacsya gigi ganavscaniensdducabaed duedleye deusas dca vieuddgadavsdavienstecyinetsteeiseidtebae tot cyvaedanboseacses 23 

Table 20: Optional Rater Perspective Item and RESPONSE OPtiOns .........sssessscsceesesssssscccesecssesnesasececsseseesaes 25 
Table 21: Pearson Product-Moment Correlations for Personnel Ratings of Administrative Roles........... 28 


Table 22: Component Matrix Coefficients for Personnel Performance Ratings of Administrative Roles 29 
Table 23: Pearson Product-Moment Correlations between Personnel Ratings of Administrative 


Roles and their Summary Judgments of the AGMINIStrAtOF .........ssccccccesecececnnescecccneseecesaeneess 29 
Table 24: Pearson Product-Moment Correlations for Personnel Ratings of Administrators’ Personal 

CHONGCEIISTICOS :soecdotetexdes scstedede ted aoeced ati veduusate otecdcetwties vested dilated iabeted elvads saittensedivesuden icteecitletetass 31 
Table 25: Rotated Component Matrix Coefficients for Personnel Ratings of Personal Characteristics 

WILK FIVEATEMS DEICCOE icstcs sakiutes stack Covtachb cece cee as at Haas bh ean veal Sacks be adit te cha ats deat aaa Fae 33 
Table 26: Pearson Product-Moment Correlations between Personnel Ratings of Personal 

Characteristics and Summary Judgments of the ACMINIStrAtOL .........cccceecccceceessecesscecesessceseenees 34 
Table 27: Pearson Product-Moment Correlations between Personnel Ratings of Factor Scale Scores 

and Summary Judgments of the ACMiINiStAtOL ..........sssesssccccceessesnscesececesesneaasecscesesnesasaeseescesesaes 34 
Table 28: Pearson Product-Moment Correlations between Factor Scale Scores and Administrator 

DEMOGI GIGS ekki cca soeie ete tea sais Fe Peau Ret cava ee see ach ca eA ob 36 
Table 29: Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary 

Judgments by the Administrator’s Perception of the Unit’s Reputation of Effectiveness ........ 37 
Table 30: Means Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary 

Judgments by the Administrator’s EXP@Ctations .....ccccccccccsesssssscececssesssssscesccscsnssnsaesessessesnesasaees 37 
Table 31: Descriptive Statistics and ANOVA Results for Ratings of 10 Administrative roles by 

Frequency of Meaningful Contact with the ACMIniStrator .........ccssccccccccesessnsssececesesesstsssesceseenes 39 


Table 32: Descriptive Statistics and ANOVA Results for Ratings of 18 Personal Characteristics by 
Meaningful Contact with the ACMiNiStrator .........:sscccccccccsessnsscscecesssnsnssscsccsssseesasaesesscssesssaeaeees 40 


Table 33: Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of 

Summary Judgments by Respondents’ POSItION ........sssessscccccecessnsscecececssesnesascescssesnesasaesesseseeenes 41 
Table 34: Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of 

Summary Judgments by Respondents’ Principal Type of Contact with the Administrator ....... 42 
Table 35: Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of 


Summary Judgments by Respondents’ Meaningful Contact with the Administrator ............... 42 
Table 36: Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Overall 

Impressions by Respondents’ Length of Time at the INStitutiOn ..........ccsssccccccssessssscesecessssentnaes 43 
Table 37: Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Overall 

Impressions by the Number of Rater Perspective Questions COMPpIeted .........scsssscccccesssessseeees 44 
Table 38: Frequency and Proportion of Times Rater Perspective Subgroups Differed from 

Administrator’s Mean by Magnitude Of |.5] .......:scccccsssssccesssseceessssecesssssecesesecesasseceessseceseseees 45 
Table 39: Estimates of Split-Half Reliabilities and Standard Errors of Measurement for Personnel 

Ratings Of AAMinistrative ROIS .......ccccscssssscccccensnssscececscesnessassececssesneaasasseesessesaaassessceseeneaaaaesenss 47 
Table 40: Estimates of Split-Half Reliabilities and Standard Errors of Measurement for Personnel 

Ratings Of Personal ChAracteristics ......csscsccccccscesesscscecscssssnsascesessesnesascesesscssesnesasssseseseesssaasasees 48 
Table 41: Estimates of Split-Half Reliabilities and Standard Errors of Measurement for Overall 

SUIMIMATY JUAGMENES ....scscccccccccssesnesssececssesnssasaesecscssessaaesesscssessenasassesseseaasaesesscsseseassseecsseseesasaes 49 
Table 42: Internal Consistency Coefficients (Cronbach @) and Item-Total Statistics for Administrator 

Hc) | e110 [Beer POS ORE TERPS TOTT PEST PES UPTPPRISTTTEPESSTPTT CHT CHT PPPIS TTT PESTERELECTUET EERE TTTE PET URTT STEPPER TTT PERLE TT OEP PERE 49 

Table 43: Internal Consistency Coefficients (Cronbach a) and Item-Total Statistics for Interpersonal 

Characteristics and Leadership Style SCOICS ........ccccceseesenssceeceensnennscesececessensaececscssnsneaasaesesssseesaes 50 
Table 44: Pearson r Correlations between Personnel Ratings of the Same Administrator on Two 

DIFFEFENE: OCCASIONS 3 s3si cscs. des deste voasccs cee s seeds coeds essa ties tesa PA 51 
Figure 1: The IDEA Center Model of Reflective Practice .........ccccccccsssssceeeessessaecececssessneceecescessususaesesesssnssaaaas 1 
Figure 2: Personnel Ratings of What Kind of Job the Administrator is doing by Frequency of 

Meaningful Contact with the ACMiniStrator .........cccccseesesssececesesseaseesecscesesnsascessceseeneaasaeseseeseeaaes 43 


INTRODUCTION 


This report explains the process, analyses, and findings that guided the revision of the IDEA 
Feedback for Administrators system. The report describes the historical background of the system, the 
revision of the instrument, validity evidence, and reliability and standard errors of measurement. 


The IDEA Feedback for Administrators system provides feedback to academic administrators 
about their performance of relevant administrative responsibilities and their leadership style and 
interpersonal characteristics. The system is based on a model of reflective practice (see Figure 1), which 
is consistent with The IDEA Center’s longstanding approach to professional development. Centra (1993) 
observed that improvement is more likely to occur when new knowledge is acquired and the individual 
is motivated to change. Administrators are probably more likely to improve if they receive useful 
feedback. 


Figure 1 
The IDEA Center Model of Reflective Practice 
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The IDEA Center makes no claim that the system results in improvement. Rather, collecting 
feedback merely begins the process, as indicated at the top of Figure 1. With the aid of explanations 
contained in the revised IDEA Feedback for Administrators Report, administrators can interpret the 
results of the Administrator Information Form-Revised (IF-R) and the Impressions of Administrator- 
Revised (IA-R) survey. As Cohen (1980) and Brinko (1991) pointed out in their reviews of research on 
student feedback to instructors, without consultation (i.e., reflection and discussion) improvement is 
unlikely. The administrator is encouraged, therefore, to discuss the results with a trusted colleague such 
as a supervisor or colleague. Improvements are more likely if administrators reflect and talk about the 
feedback. They may then design a plan to support their improvement efforts using a wide array of 
resources — from utilizing a mentor or coach to studying the many books and articles on the topics. 


The IDEA Center does not advocate high-stakes use of its instruments. As with any effective 
evaluation system, multiple sources of information should be collected, analyzed, and interpreted. 
Making a summative decision about an administrator’s employment status based solely on the results of 
the IDEA Feedback system would run counter to its intent. Many of the publications listed in the 
reference section of this document provide examples of the kinds of data one could collect as additional 
evidence of the administrator’s effectiveness (e.g., administrative portfolios). The IDEA system should 
not be used as the sole source of input for either formative or summative evaluation. However, its 
unique focus on feedback about the performance of relevant responsibilities and associated personal 
characteristics makes it an extremely useful tool for professional development purposes. 


HISTORICAL BACKGROUND 


The IDEA Feedback for Administrator System is “based on the fundamental assumption that those 
who are in a position to observe the administrator over a period of time will be able and willing to 
provide valid estimates of both performance and of style/characteristics” (Hoyt, 2004). No formal theory 
underlies the development of the items. Rather, Hoyt (2004) created a model that posits administrative 
effectiveness as being a function of (a) administrative roles, (b) personal characteristics of the 
administrator, and (c) extraneous factors that are beyond the control of the administrator. Through 
reviews of the literature and feedback from practicing administrators, he developed a comprehensive 
list of important administrative responsibilities. Using those same sources, Hoyt identified specific 
characteristics that he expected would relate to administrative effectiveness. 


In terms of administrative roles, Hoyt perceived five main foci: (a) planner, which requires 
commucating a vision and having sound priorities; (b) consultant, which requires making wise judgments 
and being an effective team player; (c) communicator, which involves communicating effectively with 
and seeking the opinion of significant others; (d) expert, as shown in knowledge and ability to anticipate 
problems before they occur; and (e) community builder, which concerns building up the institution’s 
image and earning the respect and trust of others. 


With respecto to personal characteristics, Hoyt posited six administrative styles each of which 
contributes to administrative effectiveness: (a) democratic practice (approachability, democratic 
practice, and openness to ideas); (b) structuring (organization, clarity, and predictability); (c) vigor 
(decisiveness, vigor, and activeness); (d) interpersonal sensitivity (caring, understanding, and warmth); 
(e) integrity, (honesty, fairness, and trustworthiness); and (f) character, (straightforwardness, 
consistency, and institution-centeredness). 


The IDEA Feedback for Administrators system was first made available nationally through The 
IDEA Center in 1998 to provide information for formative and/or summative evaluation. The 
administrator completed the Administrator Information Form (AIF); personnel completed the 
Impressions of Administrator (lA) survey. The AIF contained 13 questions about the administrator’s 
experience, his/her perceptions about characteristics of the unit, the general reputation of the unit, 
expectations administrators faced at the time of their appointment, the history of turnover in the unit, 
and changes that had occurred in the unit since the administrator’s appointment (see Appendix A). AIF 
responses were used for research purposes and did not influence the results provided to the 
administrator. 


On the Impressions of Administrators (IA) survey, respondents rated the administrator’s 
strengths and weaknesses relative to 10 administrative roles, using a scale from 1 (Definite Weakness) to 
5 (Definite Strength). They then rated the administrator on 18 personal characteristics, using a seven- 
point semantic differential scale. Respondents also provided ratings of confidence in the administrator 
and their perceptions of the administrator’s job performance. Finally, they answered three open-ended 
questions and responded to four demographic questions about their affiliation with the institution, 
principle contact with the administrator, frequency of meaningful contact with the administrator, and 
length of service at the institution (see Appendix A). 


Responses were initially collected using paper scanable forms. Beginning in 2001, surveys were 
administered online. 


REVISION OF THE IDEA FEEDBACK FOR ADMINSTRATORS SYSTEM 


The revisions to the of the IDEA Feedback for Administrator System was guided by statistical 
analyses performed on the 2001 to 2008 database of users, client feedback, review and consultation 
with an expert panel, and a focus group with users of the IDEA instruments. Each is described in the 
sections that follow. 


Analysis of the 2001-2008 Database 


From Spring 2001 to Summer 2008, 732 academic administrators from 58 institutions completed 
the AIF. From Spring 2002 to Summer 2008, 38,325 personnel (i.e., faculty, students, staff members, and 
other personnel) from those same 58 institutions were invited to evaluate their respective 
administrator, using the IA survey. Of those invited, 72.1% (27,632) responded. Personnel data were 
aggregated for each administrator, and analyses were performed on the aggregated data set unless 
otherwise indicated. 


Demographic Information — Administrative Units and Institutions: 

Participating institutions represented a wide range of states and regions of the U.S. Respondents 
came from 33 states, Guam, and the Marshall Islands. As indicated in Table 1, the majority (51.7%) of 
administrators came from the North Central region followed by those in the Middle State Association 
(27.7%). Size of units (i.e., number of personnel asked to respond) varied widely, from 2 to 1,735 (see 
Table 2), with an average of 60 personnel. The mean response rate was 71%. 


Table 1 
Frequencies and Percentages of Institutions 
Representing Various Accreditation Regions 


Region N % 

Middle States Assoc. of Colleges & Schools 16 927.7 
New England Assoc. of Schools & Colleges 5 8.6 
North Central Assoc. of Colleges & Schools 300 =—51.7 
Northwest Commission on Colleges & Univ. 1 1.7 
Southern Assoc. of Colleges & Schools 5 8.6 
Western Assoc. of Schools & Colleges 1 1.7 
Total 58 100 


Table 2 
Means and Standard Deviations for Unit Size and Personnel Response Rate 
Variable Minimum = Maximum M SD N? 
Number asked 


2 1735 60.26 147.64 730 
to respond 


Response Rate .10 .99 71.12 0.19 701 


Note: M= mean; SD = standard deviation 
*For 29 cases, number responding exceeded number asked to respond. 


Table 3 presents frequencies and percentages of annual student enrollments and tuitions at the 
participating administrators’ institutions across the 7-year period. Enrollments ranged from 200 to 
47,993 students, with a mean enrollment of 9,625 (SD = 9,576). Tuitions ranged from $1,404 to $30,530 
with a mean tuition of $9,531.58 (SD = $7,774.25). Participating institutions, therefore, varied 
substantially in their student enrollment and tuition. However, most enrollments ranged from 1,000 to 
10,000 students (62.9%); tuitions were almost evenly divided between those below $5,000 (49.1%) and 
those above $5,000 (50.9%). 


Table 3 
Frequencies and Percentages of Participating Administrators’ Institutional 
Enrollments and Tuitions 


Number Enrolled N % Tuition (S) N % 

< 999 4 3.8% < $4,999 52 49.1% 
1,000-—4,999 42 40.0% $5,000 — 9,999 12 11.3% 
5,000-—10,000 24 22.9% $10,000-14,999 11 10.4% 
10,000 — 14,999 7 6.7% $15,000-19,999 17 16.0% 
15,000 — 19,999 10 9.5% > $20,000 14 13.2% 
> 20,000 18 17.1% 

10 100.0 10 
Total ‘ Total 6 100.0 


Note: Tuition and enrollment data varied from year to year for those institutions that contributed data over 
multiple years. Accordingly, the enrollment and tuition values represent aggregated data over multiple 
institutions across multiple years. Therefore the total Ns (106/105) do not represent the total number of 
institutions (58) but the total number of data points (institutions by years). 


Demographic Information — Administrators and their Units: 

Table 4 displays frequencies and percentages of ranges in administrators’ years of service at the 
institution, in the current position, and in the same position at any institution. Most administrators had 
been at their institution more than 5 years (68.2%). The majority had been in their current position 5 or 
fewer years (61.5%) and in that position at any institution 5 or fewer years (52.7%). Because years of 
service at the institution (Wdn = 10, M = 12.86, SD = 9.94) and in the current position (Wdn =5, M= 
5.87, SD = 4.96) were positively skewed, the medians of 10 and 5 years, respectively, best represent the 
average terms of service. Including interim leadership and themselves, the average number of 
individuals who had occupied the administrators’ position during the past 10 years, was 2 (M = 2.30, 
Mdhn = 2.00, SD = 1.70). 


Table 4 
Frequencies and Percentages of Ranges in Administrators’ Years of Service 
This Position any 


Number At Institution In Current Position iaceeUGH 

of Years N % N % N 9% 

<1 3 0.4 10 1.4 64 8.9 
1to5 229 31.4 437 60.1 315 43.8 
6 to 10 152 20.9 179 24.6 184 25.6 
11 to 15 88 12.1 52 7.2 63 8.7 
16 to 20 81 11.1 34 4.6 44 6.1 
21 to 25 68 9.3 14 2.0 27 3.7 
26 to 30 53 7.3 0 0.0 7 1.0 
> 30 55 7.5 1 0.1 16 2.2 
Total 129 100.0 727 100.0 220 100.0 
NR 3 0.4 5 0.7 12 1.6 


Note: NR = Not Responding 


Table 5 presents frequencies and percentages of the number of professionals and support staff 
(omitting students) working in the units. Most administrators (65.5%) reported that support staff 
numbered fewer than 10 individuals; 51.3% reported the same with respect to professional staff. 
Because the number of support staff (Wdn = 5, M = 33.70, SD = 171.69) and professionals in the unit 
(Mdn = 9, M = 58.21, SD = 216.08) were positively skewed, the medians of 5 and 9 professionals, 
respectively, may best represent the average number of personnel. 


Table 5 
Frequencies and Percentages of Numbers of Support Staff 
And Professionals in the Unit 


Number of Support Staff Professionals 
Staff N % N % 

< 10 449 65.5 352 51.3 
10 to 19 88 12.9 102 14.9 
20 to 29 42 6.1 60 8.7 
30 or more 106 15.5 172 25.1 
Total 685 100.0 686 100.0 
NR 47 6.4 46 6.3 


Demographic Information — Respondents: 

From Spring 2002 to Summer 2008, 38,325 personnel (i.e., faculty, students, staff members, 
and others) were invited to complete the IA survey. Of these, 27,632 individuals responded, resulting in 
an overall response rate of 72.1% (SD = 0.19; Med = .74). Table 6 presents frequencies and percentages 
of various personnel characteristics. Excluding those who chose not to respond, the majority of 
respondents (55.5%) were faculty members (see Item 34). The principal type of contact personnel had 
with the administrator varied substantially (Item 35). Half of the respondents (50.4%) met with the 
administrator anywhere from daily to at least one-to-three times per month (Item 36). Most (67.2%) had 


been in their current institution for more than five years (Item 37). In general, then, most respondents 
were experienced and met with the administrator fairly frequently. 


Table 6 
Frequencies and Percentages of Various Personnel Characteristics 
Respondent Characteristics N Percent 
34, What is the principal way you are affiliated with this institution? 
lam a faculty member. 13,151 55.5% 
lam a student. 681 2.9% 
lam amember of this administrator's staff. 4,534 19.1% 
| have meaningful contacts with this administrator but don't report to 
this person. 5,323 22.5% 
Total 23,689 100.0% 
Chose not to respond 3,943 14.3% 


35. What is the principal type of contact you have with this administrator? 


We work together on mutual responsibilities or assignments. 3,869 18.7% 
My work requires his/her supervision, approval, or judgment. 5,896 28.4% 
This administrator provides services | want or need. 4,945 23.9% 
Our contact consists primarily of informal interaction. 6,022 29.0% 
Total 20,732 100.0% 
Chose not to respond 6,900 25.0% 
36. How often do you have meaningful contact with this administrator? 
Once a week or more 6,301 27.1% 
One to three times a month 5,396 23.3% 
One to three times a term 5,669 24.5% 
Less than once a term 5,831 25.1% 
Total 23,197 100.0% 
Chose not to respond 4,435 16.1% 


37. How long have you been at this institution? 


This is my first year 1,348 5.7% 
One to two years 2,109 9.0% 
Three to five years 4,253 18.1% 
More than 5 years 15,792 67.2% 
Total 23,502 100.0% 
Chose not to respond 4,130 14.9% 


Note: Percentages based upon total Ns excluding missing values. 


Correlations between Number of Personnel Asked to Respond, Response Rates, and Ratings of the 
Administrator: 

Table 7 displays Pearson r correlation coefficients (with personnel ratings aggregated for each 
administrator) between the respondent ratings and two variables: number of personnel asked to fill out 
a form and the overall response rate for each administrator. Although the directions of the relationships 
varied, response rate explained approximately 2% to 14% (rs ranging from |.14| to |.38]) of the 
variance in ratings for several items. For administrative roles (Items 1 to 10) and personal characteristics 
(Items 11 to 28), ratings increased as response rate increased. In addition, overall impressions of the 


kind of job the administrator was doing (Item 29) correlated positively with response rate. Explanations 
for these observed relationships are numerous. Personnel who held favorable impressions of their 
administrator may have been more likely to respond to the survey. When an administrator was 
performing well individuals were perhaps more enthusiastic about evaluating him or her, and they 
responded accordingly. High response rates might also suggest that the administrator (or his/her 
supervisor) carefully identified individuals best able to respond to the survey, which is another indicant 
of an effective administrator. Alternatively, some administrators may have been less selective and 
invited individuals who were less likely to respond. Regardless of the explanation, the correlations in 
Table 7 indicate response rate is more important than the number of individuals asked to respond. 
However, the correlations are not high enough to warrant great concern. In addition, the high overall 
response rate (72.1%) within the population of personnel invited to participate lessens concern about 
these relationships. Nonetheless, we will return to this issue in a subsequent section of this report when 
we compare respondents and non-respondents to the rater perspective questions (Items 34 to 37 on 
the IA survey). 


We also investigated whether response rate was related to the number of individuals 
administrators asked to respond. The correlation was moderate in strength and negative in direction, r= 
-.44. So, as the number of individuals asked to respond increased, the response rate decreased. This 
speaks to the issue raised in the preceding paragraph about careful selection of raters. Identifying a 
wide range of potential respondents without concern about their likelihood of responding could 
contribute to lower response rates. Because response rate affects representativeness, all 
administrators, but especially those in larger units, should be encouraged to increase their efforts at 
improving personnel response rates. In addition, care should be taken in identifying respondents who 
feel qualified to evaluate the administrator. Those who feel unqualified are probably less likely to 
respond. 


Table 7 


Correlations between Respondent Ratings, Number Asked to Respond, and Response Rate (personnel 


responses aggregated for each administrator, N = 732) 


r with rwith 
a pa fairey Number Response 
(Administrator level data) Asked paid 
1. Communicates and displays a visionary plan for the future. -.11 31 
2. Establishes sound priorities. -.14 33 
3. Displays knowledge/expertise required for this position. -.18 38 
4. Makes wide decisions, judgments, and recommendations. -.17 36 
5. Initiates actions that anticipate problems or resolves them. -.16 31 
6. Is an effective "team" member. -.13 32 
7. Contributes positively to this institution’s image and reputation. -.16 38 
8. Communicates relevant information to appropriate constituencies. -.15 31 
9. Seeks opinions of others before establishing policies or procedures. -.17 36 
10. Earns the trust and respect of those who come in contact with him/her. -.13 32 
11. Indecisive vs. Decisive. -.11 21 
12. Disorganized vs. Organized -.05 14 
13. Remote vs. Approachable. -.17 35 
14. Untruthful vs. Honest -.15 33 
15. Unfair vs. Fair -.16 33 
16. Autocratic vs. Democratic -.13 .27 
17. Unfeeling vs. Caring -.17 35 
18. Manipulative vs. Straightforward -.12 .28 
19. Inconsistent vs. Consistent -.09 19 
20. Lethargic vs. Vigorous -.13 31 
21. Ambiguous vs. Clear -.10 .24 
22. Self-centered vs. Institution-centered -.11 33 
23. Insensitive vs. Understanding -.17 35 
24. Opinionated vs. Receptive to ideas -.13 .27 
25. Untrustworthy vs. Trustworthy -.15 33 
26. Passive vs. Active -.13 .28 
27. Aloof vs. Warm -.15 34 
28. Erratic vs. Predictable -.09 17 
29. What kind of a job is this administrator doing? -.16 37 
30. Confidence in administrator -.01 35 


Note: Positively worded items (1 = positive characteristic) were reverse-coded so that higher values for every 


personal characteristic represented more positive evaluations of the administrators. 


Administrator Perceptions of the Unit: 

Administrators responded to several questions about their perceptions of the unit’s 
characteristics and reputation. Only about 1 in 5 (22.2%) administrators perceived a negative campus 
reputation regarding their unit’s importance/influence when they first assumed the position (see Item 4, 
Table 8). A slightly greater percentage (27.7%) perceived a negative reputation about the unit’s 
effectiveness (Item 5). Most (65.9%) were expected to make major changes in the unit’s activities when 
they took the position (Item 6). About half (51.5%) of the respondents estimated the turnover rate in 
the unit was about average before they assumed the current position (Item7). From the perspectives of 
the administrators, then, their units had, for the most part, experienced fairly stable leadership and had 
enjoyed good campus reputations and average turnover; nonetheless, the administrators believed they 
were expected to make changes. 


As indicated in Table 8 (Item 10), most administrators (88%) believed the workload in the unit 
had increased since their initial appointment. Three out of four (81%) reported financial support for the 
unit either had remained constant or increased relative to that of other units (Item 9). About half 
(54.1%) reported the turnover rate had remained constant since they had been in the position (Item 8). 


Table 8 
Frequencies and Percentages of the Administrator’s Perceptions of the Unit’s Characteristics 


Item in AIF N % 


4, When you first assumed this position, what was the general campus reputation of your unit in 
terms of importance/influence: 


Positive 268 38.8 
Neutral 171 24.8 
Negative 153 22.2 
NA/new unit 59 8.5 
Don't know 39 5.7 
Total 690 100.0 
Chose not to respond 42 5.7 


5. When you first assumed this position, what was the general campus reputation of your unit in 
terms of effectiveness: 


Positive 201 29.1 
Neutral 195 28.3 
Negative 191 27.7 
NA/new unit 59 8.5 
Don't know 44 6.4 
Total 690 100.0 
Chose not to respond 42 5.7 
6. When you first assumed this position, you were expected to: 
Make major changes in the unit's activities 454 65.9 
Primarily maintain established services and procedures 181 26.3 
Other 54 7.8 
Total 689 100.0 
Chose not to respond 43 5.9 
7. Before you assumed this position, turnover of personnel in the unit had been: 
Higher than most units 84 12.2 
About average 355 51.5 
Lower than most units 118 17.1 
Don’t know 112 16.3 
I’d prefer not to respond 20 2.9 
Total 689 100.0 
Chose not to respond 43 5.9 
8. Since you have been in this position, personnel turnover has: 
Increased 83 12.1 
Remained constant 372 54.1 
Decreased 161 23.4 
Don’t know 56 8.1 
I’d prefer not to respond 16 2.3 
Total 688 100.0 
Chose not to respond 44 6.0 


Continued on next page: 
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Table 8: continued 
9. Since you have been in the position, financial support for the unit, relative to that for other units 
at the institution has: 


Increased 314 45.5 
Remained constant 245 35.5 
Decreased 84 12.2 
Don't know 38 5.5 
I'd prefer not to respond 9 1.3 
Total 690 100.0 
Chose not to respond 42 5.7 
10. Since you have been in this position, the workload of the unit has: 
Increased 607 88.0 
Remained constant 70 10.1 
Decreased 4 6 
Don't know 7 1.0 
I'd prefer not to respond 2 3 
Total 690 100.0 
Chose not to respond 42 5.7 
Summary: 


Typically, administrators had served an average of 5 years in their current position. Most had 
been at their institution more than 5 years. The median number of individuals that had occupied the 
administrator’s position during the previous 10 years was 2. An average of 5 support staff and other 
professionals served within the administrator’s unit. Most personnel who responded to the survey were, 
in general, experienced individuals that met with the administrator fairly frequently. For the most part, 
administrators believed they faced positive conditions upon initial appointment. About 1 in 5 believed 
their unit’s campus reputation of importance and influence was negative at the time they assumed the 
position; approximately 1 in 4 perceived a negative campus reputation about their unit’s effectiveness at 
the time of appointment. Nonetheless, most administrators believed they were expected to make major 
changes in the unit’s activities. The vast majority indicated the workload in the unit had increased since 
their initial appointment, and 3 out of 4 reported financial support for the unit had remained constant 
or increased relative to that of other units at their institution. About half of the administrators indicated 
turnover rates were about average at the time of their appointment, and about the same proportion 
indicated rates had remained constant. 


Analysis of 2001-2008 Database: Descriptive Statistics 


Personnel Ratings of the Administrator’s Performance of Administrative Roles: 

The first section of the survey asked respondents to rate the administrator’s strengths and 
weaknesses with respect to 10 administrative roles. Respondents used the scale of 1 = Definite 
Weakness, 2 = More a Weakness than a Strength, 3 = In Between, 4 = More a Strength than a Weakness, 
5 = Definite Strength, and CJ = Cannot Judge. Table 9 shows the percentage of administrators falling 
within six categories of mean ranges for each item on the 1 to 5 scale. Table 10 presents means and 
standard deviations for each item. Notably, 44.7% of administrators were in the highest range category 
(4.50-5.00) on “Displays knowledge/expertise required for this position” (Item 3), and 49.0% were in the 
highest range on “Contributes positively to this institution’s image/reputation” (Item 7). Substantially 
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fewer administrators (15.4% and 14.5%, respectively) received the highest rating on “Communicates and 
displays a visionary plan for the future” (Item 1) and “Initiates actions that anticipate problems or 
resolves them before they become major concerns” (Item 5). In general, then, personnel perceived their 
administrators as having appropriate knowledge and expertise and contributing positively to the 
institution’s reputation. They were relatively less enthusiastic about their administrators’ vision and 
initiative, although the overall means for these roles indicated personnel perceived them as more 
strengths than weaknesses (see Table 10). 


Table 9 
Percentage of Administrators Falling Within Mean Category Ratings of 10 Administrative Roles (N = 732) 
Ranges of Mean Ratings and 


Percentage of Administrators Falling Within Each Category 
1.00- 2.00- 2.50- 3.00- 3.50- 4.00- 4.50- 
1.99 2.49 2.99 3.49 3.99 4.49 5.00 


Administrative Roles 


1. Communicates and displays a 

visionary plan for the future. 

2. Establishes sound priorities. 0.4 0.8 3.0 14.1 24.5 38.8 18.4 

. Displays knowledge/expertise 
required for this position. 

4. Makes wise decisions, 
judgments, and 0.4 1.0 2.6 13.6 22.3 38.5 21.6 
recommendations. 

. Initiates actions that anticipate 
problems or resolves them. 


0.3 0.9 4.7 14.2 25.8 38.7 15.4 


WW 


0.3 0.2 1.0 5.6 15.6 32.6 44.7 


uw 


0.4 1.5 5.6 15.3 27.5 35.2 14.5 


6. Is an effective "team" member. 0.3 1.6 3.8 11.4 21.2 30.4 31.3 
7. Contributes positively to this 
institution’s image and 0.0 0.4 1.4 5.4 14.4 29.4 49.0 
reputation. 
8. Communicates relevant 
information to appropriate 0.4 0.4 2.6 13.0 23.8 35.8 24.0 


constituencies. 
9. Seeks opinions of others before 


establishing policies or 0.5 2.0 6.0 15.7 25.0 29.5 21.3 
procedures. 

10. Earns the trust and respect of 
those who come in contact with 0.3 1.5 4.1 12.7 19.8 31.4 30.2 
him/her. 


Note: Number in cells represents % of administrators to receive a mean rating within the specified response range. 
Total N Administrators = 732. Total N Respondents = 27,632. Personnel responded to all items on a1 
(Definite Weakness)-to-5 (Definite Strength) scale. 
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Table 10 
Means and Standard Deviations for A-Priori Administrative Role Subscales and Items (N = 732) 


Administrative Scales and Items Mean SD 

Planner 3.96 0.53 
1. Communicates and displays a visionary plan for the future. 3.94 0.56 
2. Establishes sound priorities. 3.99 0.54 
Consultant 4.06 0.58 
4. Makes wise decisions, judgments, and recommendations. 4.03 0.56 
6. Is an effective "team" member. 4.09 0.63 
Communicator 3.98 0.59 
8. Communicates relevant information to appropriate constituencies. 4.05 0.56 
9. Seeks opinions of others before establishing policies or procedures that affect 391 065 
them. : : 

Expert 4.10 0.52 
3. Displays knowledge/expertise required for this position. 4.31 0.51 
5. Initiates actions that anticipate problems or resolves them before they 389 058 
become major concerns. , : 

Community Builder 4.21 0.56 
7. Contributes positively to this institution’s image and reputation. 4.35 0.51 
10. Earns the trust and respect of those who come in contact with him/her. 4.08 0.63 
Total 10-Item Scale 4.06 0.58 


Note: Means and SDs were computed by calculating an average response per item after aggregating respondent 
data for each administrator. Personnel responded to all items on a 1 (Definite Weakness) to 5 (Definite 
Strength) scale. 


Ratings of the Administrator’s Personal Characteristics: 

Respondents rated their administrator on 18 personal characteristics (Items 11 to 28), using a 7- 
point semantic differential scale. Because the direction of the scale varied (for half the items a “1” was 
associated with a positive characteristic; for the other half a “1” was associated with a negative 
characteristic), items were re-coded prior to analyses so that higher values for every personal 
characteristic represented more positive evaluations of the administrator. Means and standard 
deviations for each personal characteristic, organized by a-priori administrative style subscales, can be 
found in Table 11. Personnel gave administrators the highest rating on the Integrity subscale and the 
lowest on Democratic Practice, although differences among subscales were not large. 
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Table 11 
Means and Standard Deviations for A-Priori Administrative Style Subscales and Items (N = 732) 


Administrative Style Scales and Items Mean SD 
Democratic Practice 5.30 0.90 
13. Remote vs. Approachable 5.42 0.86 
16. Autocratic vs. Democratic 5.04 0.99 
24. Opinionated vs. Receptive to ideas 5.45 0.85 
Structuring 5.54 0.72 
12. Disorganized vs. Organized 5.63 0.74 
21. Ambiguous vs. Clear 5.43 0.75 
28. Erratic vs. Predictable 5.54 0.68 
Vigor 5.70 0.67 
11. Indecisive vs. Decisive 5.62 0.65 
20. Lethargic vs. Vigorous 5.78 0.66 
26. Passive vs. Active 5.70 0.70 
Interpersonal Sensitivity 5.62 0.78 
17. Unfeeling vs. Caring 5.70 0.73 
23. Insensitive vs. Understanding 5.73 0.74 
27. Aloof vs. Warm 5.44 0.86 
Integrity 5.87 0.73 
14. Untruthful vs. Honest 5.92 0.69 
15. Unfair vs. Fair 5.75 0.77 
25. Untrustworthy vs. Trustworthy 5.94 0.75 
Character 5.67 0.77 
18. Manipulative vs. Straightforward 5.64 0.81 
19. Inconsistent vs. Consistent 5.56 0.73 
22. Self-centered vs. Institution-centered 5.81 0.77 
Total 18-Item Scale 5.62 0.77 


Note: Means and SDs were computed by calculating an average response per item after aggregating respondent 
data for each administrator. SD = Standard deviation. Personnel responded to all items on a1 to 7 
semantic differential scale. 


Summary Judgments of the Administrator: 

Respondents answered two questions designed to assess their summary judgments of the 
administrator’s effectiveness. The Pearson r correlation between the two items was r= .92. Item 29 
asked, “What kind of a job is this administrator doing?” Response options were: 1 = Poor, 2 = Mediocre, 
3 = Good, 4 = Excellent, 5 = Superb and CJ = Can’t Judge (designated as system missing). Overall, 
administrators were viewed as doing a “close-to-excellent” job (M = 3.72, SD = 0.59, N = 732, see Table 
12). 


Item 30 asked, “Does this administrator have your confidence?” It offered five options: 1 = 
Definitely not, 2 = No, but I have reservations about this, 3 = Yes, but | have reservations about this, 4 = 
Definitely yes, and 5 = 1 have formed no stable opinion. For all analyses, the value of 5 was set to system 
missing. Overall, respondents had confidence in their administrators (M = 3.50, SD = 0.43, N = 732). 
Percentages of personnel who responded in each category are presented in Table 13. 
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Table 12 
Percentages of Administrators Falling Within Ranges of Category Ratings on Item 29 (N = 732) 


Ranges of Mean Ratings and Percent of Administrators Falling Within Each Category 


Sat cede tid ts cena aoe 2.00- 2.50- 3.00- 3.50- 4.00- 4.50- Mean SD 
8 P oe 1.99 2.49 2.99 3.49 3.99 4.49 5.00 
£o2 vena SINLene bis: Fie aaministiatel 0.1 0.6 1.8 9.2 21.8 29.9 28.4 8.2 372). “0.59 


doing? 


Note: Means and SDs were computed by calculating an average response per item after aggregating respondent data for each administrator. Number in cells 
represents % of administrators to receive a mean rating within the specified response category range. Total N Administrators = 732. Total N 
Respondents = 27,632. Personnel responded using the scale of 1 = Poor, 2 = Mediocre, 3 = Good, 4 = Excellent, 5 = Superb. 


Table 13 
Percentages of Administrators Falling Within Ranges of Category Ratings on Item 30 (N = 732) 


Ranges of Mean Ratings and Percent of Administrators Falling Within 


Each Category pia SD 
Rating of Overall Impression 1.0-1.49 1.50-1.99 2.00-2.49 2.50-2.99 3.00-3.49 3.50-4.00 
30. Does this administrator have your 03 0.4 50 93 16.3 52.4 3.50 0.4 
confidence? 3 


Note: Means and SDs were computed by calculating an average response per item after aggregating respondent data for each administrator. Number in cells 
represents % of administrators to receive a mean rating within the specified response category range. Total N Administrators = 732. Total N 
Respondents = 27,632. Personnel responded using the scale of 1 = Definitely not, 2 = No, but | have reservations about this, 3 = Yes, but | have 
reservations about this, 4 = Definitely yes. 
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Summary: 

Personnel rated their administrators highest on displaying knowledge/expertise, making positive 
contributions to the institution’s image/reputation, and integrity. Administrators received relatively 
lower ratings on communicating a vision, resolving problems before they become major concerns, and 
democratic practice. Overall, personnel expressed confidence in their administrators and thought they 
were doing a “close-to-excellent” job. 


User and Expert Review 


Input from Focus Group: 

The IDEA Center staff conducted a focus group during the 2008 Academic Chairperson’s 
Conference held in Orlando, Florida. Six individuals participated (4 women and 2 men); three had 
participated as administrators and three as raters. The following protocol was used for the focus group. 


Introduction (5-10 minutes) 
e Very briefly, give your reaction to the feedback tool (Positive, Negative, or Neutral). 
In a sentence or two please give your reasons for that reaction. 


The Survey 

e What are the strengths of the instrument? Weaknesses? 

e In what ways would you modify the instrument? Which of your recommendations 
do you believe is most important? 

e What specific recommendations do you have with regard to content, additional 
items, or items to delete? 


The Administrator Information Form 

e In what ways would you modify the instrument? Which of your recommendations 
do you believe is most important? 

e What specific recommendations do you have with regard to content, additional 
items, or items to delete? 


The Use of Norms 

e How prominent should the reporting of norms be in the report? 

e There are competing opinions about the value of norms. How important would it 
be to you, or your dean, to see normative information? 


Audience 

e This instrument allows for a broad audience to participate (e.g., students, faculty 
staff, and colleagues). Should a revision include other audiences for which 
subgroup information might be useful (see Section V. Rater Perspective)? 


The Report 


e How might the report be better designed to provide a more coherent summary and 
recommendations for improvement? 
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e Information is conveyed using a number of methods. Let’s briefly look at each page 
of the report and see if there are any that you find more useful than others or 
particularly confusing? What other ways would you like to see information 
reported? 


e Related to the questions about audience, it has been recommended that the 
administrator provide a self-evaluation using the same items and to compare the 
administrator’s response to that of his/her faculty. Your reaction? Other 
possibilities (i.e., include supervisory evaluation)? 


Additional Resources 

e The research on student ratings indicates that for significant improvements in 
teaching to occur, there need to be resources available to guide improvement 
efforts. Examples of these resources include individual consultations, 
workshops/seminars, webinars, articles and books). What resources do you believe 
the IDEA Center might provide that would be most useful in supporting the 
improvement of dean functioning? 


Appendix C contains common themes that emerged from the sessions. The following focus 
group recommendations were incorporated into the revised feedback system and are further detailed in 
the Item Development and Process Modification sections that follow. 


1. Give administrators the option of adding additional questions. 

2. Provide the option of including “position specific” questions or the ability to select from a set of 
pre-defined questions. 

3. Of the personal characteristics, “Trustworthy” is most important. Evaluate the other personal 

characteristics to identify which ones are most important. 

Examine and perhaps revise IA Items 29 and 30 and their anchors. 

Reword IA Item 31 as “What are this administrator’s strengths?” 

Reword IA Item 32. 

Evaluate the usefulness of all rater perspective questions. 

Evaluate the usefulness of AIF Items 3, 4, and 7 through 10. 


fo De 


Input from Expert Panel: 

A panel of experts’ was invited to review The IDEA Feedback for Administrators System and 
provide input. The panel was comprised of members of The IDEA Center Board of Directors who are 
current or former higher education administrators. Some of them have written highly regarded books 
on faculty development/evaluation and higher education administration. In some cases, we requested 
and received independent feedback from individuals; at other times, meetings were held at a 
professional conference to collect additional input. The process of seeking input from these sources 
went through several stages, with revisions to items done at each step and described in the next section, 
Item Development. 


' The panel of experts was comprised of IDEA Board members who had expertise in the field of higher education 
administration. Many had written books and articles, and all had served in an administrative capacity in a college 
or university. 
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Item Development 


The initial development of the items came from the Evaluation of Administrative Style and 
Effectiveness (EASE, see Appendix D) instrument created by Donald Hoyt for evaluating administrators at 
Kansas State University in the early 1990s. The survey instrument included questions about the rater’s 
characteristics, attributes of the administrator, administrative roles, overall impressions of the 
administrator, and questions for open-ended comments. It included 26 structured questions and 4 
“free-response” questions. As indicated in the /nterpretive Guide to Ratings of Administrative 
Effectiveness (Appendix D), the EASE was not intended to be comprehensive: 


The questions are not equally relevant to all administrative positions. Furthermore, some 
positions may require activities or attributes which were omitted from this survey (e.g., Is the 
individual’s work done accurately? Is the individual creative? Can the individual be trusted?) It is 
not possible to cover every important feature of every administrative position; and the key 
requirements or attributes may vary from year to year and with different circumstances (p. 4). 


Similarly, the IA and IA-R are not comprehensive to all administrative positions, which is why the 
user has the option to include additional position-specific questions. To obtain a more comprehensive 
assessment of an administrator’s effectiveness one would need to conduct a “broader, more diverse 
inquiry” that would include, among other evidence, interviews with key constituents and staff members 
(Interpretive Guide to Ratings of Administrative Effectiveness, p. 5). Nonetheless, the IA-R is designed to 
collect useful feedback about impressions of the administrator’s effectiveness, interpersonal 
characteristics, and leadership style. 


The following sections describe the specific survey changes from the original EASE, to the 
Impressions of Administrator survey (IA) offered nationally since 1998, to the final revised Impression of 
Administrator survey (IA-R), made available beginning in 2013. The decisions to delete, modify, or retain 
an item was based on the processes described above (analysis of the 20011-2008 data, input from client 
focus group, input from expert panel, review of relevant literature,” input from IDEA staff members). 


The Administrator’s Roles: 

The IA-R asks personnel to rate the administrator on 10 roles associated with administrative 
effectiveness. The wording of the items underwent several transformations. In developing the 10 items 
for the IA, Hoyt initially proposed the following five general roles for administrators: 

1. Planner. The administrator must communicate a vision and set priorities for the unit. 

2. Consultant. Effective administrators are “team” players who make wise decisions and offer 
helpful recommendations. 

3. Communicator. Successful administrators seek the input of those who will be affected by 
policies and procedures; they communicate relevant information to appropriate constituencies. 

4. Expert. Administrators should be knowledgeable; they should anticipate potential problems 

5. Community builder. Effective administrators contribute positively to the institution’s image and 
earn the trust and respect of others. 


Each of the 10 administrative roles on the IA was associated with one of these general 
categories. The results of principal components analysis (described in a later section of this report) 


> For example: Bogue, 1994; Kouzes & Posner, 2000; McDade, 1988; Seldin, 1988; Seldin & Higgerson, 2002; Wilcox 
& Ebbs, 1992 
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indicated, however, that a single dimension underlies all 10 items. Therefore, the five general categories 
were not retained in the revised Administrator Report. 


As shown in Table 14, of 10 administrative roles on the IA-R, seven are modified versions of the 
original EASE items. Table 14 shows the progression in modifications of wording of items retained in the 


IA-R. 

Table 14 

Progression of Wording of Administrative roles Retained in the IA-R 

EASE Wording IA Wording IA-R Wording 

Communicates and displays a 1. Communicates and displays a 1. Communicating a visionary plan 


visionary and futuristic plan 


visionary plan for the future. 


for the future 


Has developed sound priorities 


. Establishes sound priorities. 


2. 


Establishing sound priorities 


Displays knowledge/expertise 
related to work assignments 


. Displays knowledge/expertise 


required of this position. 


3. 


Displaying knowledge/expertise 
required of this position 


Makes wise decisions 


. Makes wise decisions, judgments, 


and recommendations. 


4. 


Making wise decisions, 
judgments, and 
recommendations 


Initiates actions which resolve 
problems in a timely manner 


. Initiates actions that anticipate 


problems or resolves them 
before they become major 
concerns. 


. Initiating actions that anticipate 


problems or resolving them 
before they become major 
concerns. 


. Is an effective “team” member. 


. Being an effective “team” 


member 


Contributes positively to the 
institution’s image and reputation 


. Contributes positively to the 


institution’s image and 
reputation. 


. Communicates relevant 


information to appropriate 
constituencies. 


. Contributing positively to the 


institution’s image 


. Communicating relevant 


information to appropriate 
constituencies 


Seeks opinions of others before 
suggesting policies or procedures 
which affect them 


. Seeks opinions of others before 


establishing policies or 
procedures that affect them. 


. Seeking opinions of others before 


establishing policies or 
procedures that affect them 


10. Earns the trust and respect of 


others that come into contact 
with him/her. 


10. Earning the trust and respect of 


others 


The Administrator’s Leadership Style and Personal Characteristics: 
The IA-R contains 15 semantic-differential items designed to assess leadership style and 
interpersonal characteristics. Six of the items are reworded descendants of items contained on the 


EASE, seven are modified or taken directly from the IA, and two are new. 


Based on the literature cited in the previous section and feedback from faculty and 
administrators, Hoyt initially proposed several dimensions of administrative style and personal 
characteristics relevant to effective administration. He hypothesized that each was made up of three 
distinct characteristics, as described in Table 15. The initial IDEA Feedback for Administrators Report 
advised that while high ratings were generally preferred on administrative style and interpersonal 
characteristics, effective administrators could develop unique styles that departed from this 
expectation. The same recommendation applies to the 15 items that remain on the IA-R. 
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Table 15 
Administrative Style and Personal Characteristics Dimensions ° 


e Approachability, 
Democratic Practice e Democratic practice 
Receptiveness to ideas 


e Organization 
Administrative Style Structuring e = Clarity 
e = Predictability 


e Decisiveness 


Vigor e Vigor 
e ©Aggressiveness 
e Caring 
Interpersonal Sensitivity e Understanding 
e Warmth 
e Honesty 
Personal Characteristics Integrity e Fairness 


e Trustworthiness 


e =©Straightforwardness 
Character e Consistency 
e = Institution-centered 


* Used in 1998 version. 


Of the 18 items developed for the IA to assess administrator personal characteristics, 13 remain 
in the IA-R (see Table 15). Two new characteristics were added based on research demonstrating their 
validity and reliability found in the IDEA Feedback for Chairs System: flexibility and practical judgment. 
Faculty who rated their department chairs high on these two characteristics also gave high marks for 
establishing trust. Ratings of the chair’s flexibility and practical judgment were also highly correlated 
with ratings of the chair’s overall performance. The traits of flexibility and practical judgment also 
showed high inter-rater agreement and strong stability across ratings of the same chairs on more than 
one occasion (see Benton, Gross, Pallett, Song, and Webster, 2010). 


The results of principal components analysis, detailed in a later section of this report, confirmed 
the existence of two underlying dimensions among the 13 items retained on the IA-R, which we called 
Leadership Style (LS) and Personal Characteristics (PC). Items associated with these two dimensions are 
indicated by their acronyms in Table 16 under “IA-R Wording.” 
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Table 16 


Progression of Wording of Leadership Style and Personal Characteristics Items in the IA-R 


EASE Wording 


IA Wording 


IA-R Wording 


11. Decisive vs. Indecisive 


11. Indecisive vs. Decisive (LS) 


Organized vs. Disorganized 


12. Organized vs. Disorganized 


12. Disorganized vs. Organized (LS) 


13. Remote vs. Approachable 


13. Remote vs. Approachable (PC) 


15. Unfair vs. Fair 


14. Unfair vs. Fair (PC) 


Democratic vs. Autocratic 


16. Autocratic vs. Democratic 
18. Manipulative vs. Straightforward 


15. Autocratic vs. Democratic (PC) 
16. Manipulative vs. Straightforward 
(PC) 


19. Inconsistent vs. Consistent 


17. Inconsistent vs. Consistent (LS) 


Structured vs. Ambiguous 


21. Ambiguous vs. Clear 


18. Ambiguous vs. Clear (LS) 


Dedicated to institution vs. 
Dedicated to self 


22. Self-centered vs. Institution- 
centered 


19. Self-centered vs. Institution- 
centered (PC) 


Sensitive to others vs. Insensitive to 
others 


23. Understanding vs. Insensitive 


20. Insensitive vs. Understanding 
(PC) 


Open to ideas vs. Close minded 


24. Receptive to ideas vs. 
Opinionated 


21. Opinionated vs. Receptive to 
ideas (IC) 


25. Trustworthy vs. Untrustworthy 


22. Untrustworthy vs. Trustworthy 
(PC) 

23. Unimaginative vs. Innovative 
(LS) 


24. Rigid vs. Flexible (LS — 
hypothesized) 


25. Impractical vs. Practical (LS — 
hypothesized) 


26. Passive vs. Active 


14. Honest vs. Untruthful 


17. Caring vs. Unfeeling 


Energetic vs. Lethargic 


20. Lethargic vs. Vigorous 


27. Aloof vs. Warm 


28. Predictable vs. Erratic 


Leadership Style (LS) and Personal Characteristics (PC) 


Overall Impressions of the Administrator: 
Another change in the IA-R addressed the wording of two summary judgment items. On the IA, 
respondents rated the question “What kind of job is this administrator doing,” using the five-point scale 
of Poor (1), Mediocre (2), Good (3), Excellent (4), and Superb (5). Panel experts expressed two chief 
concerns about this item. First, in many upper-level administrative positions (e.g., president, chief 
academic officer), members of governing boards do not necessarily seek a “thumbs-up-thumbs-down” 
recommendation. Second, the scale made it possible for an administrator to fall in the middle of the 
distribution yet still earn a rating of “Good.” Consequently, for the IA-R, the item was reworded to 
“Overall, this administrator has provided excellent leadership.” The scale was replaced with Strongly 
Disagree (1) to Strongly Agree (5) (see Table 17). 


The second IA summary judgment item asked, “Does this administrator have your confidence” ? 
Personnel could respond with Definitely not (1); No, but | have reservations about this (2); Yes, but | have 
reservations about this; and Definitely yes (4). Again, panel experts expressed concern that this item 
would not be appropriate for evaluating administrators in upper-level positions; the panel also believed 
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the scale was restricted in range. Accordingly, for the IA-R, the item was changed to read “I have 

confidence in the administrator’s ability to provide future leadership in this position.” Again, the scale 
was changed to Strongly Disagree (1) to Strongly Agree (5). Table 17 shows the progression in wording 
of these items from the EASE to their current form. 


Table 17 


Progression in Wording of Overall Impression Items 


EASE Wording 


1A Wording 


IA-R Wording 


Overall, how effective do you 
think this administrator is? 

e = Exceptional 

e Very good 

e Good 

e = Satisfactory 

e Needs improvement 


29. What kind of job is this 
administrator doing? 


e Poor 

e Mediocre 
e Good 

e = Excellent 
e Superb 


e §6Can’t judge 


27. Overall this administrator has 
provided excellent leadership. 

e §©Strongly Disagree 

e =©Disagree 

e  =In Between 

e §6Agree 

e §=©Strongly Agree 


Does this administrator have 
your confidence? 

e §=6Always 

e ~=Usually 

e = Half the time 

e —_ Less than half the time 


30. Does this administrator have your 
confidence? 
e = Definitely not 
e No, but! have reservations 
about this. 
e §=6Yes, but | have reservations 
about this. 
e = Definitely yes 
e  |have formed no stable 
opinion 


26. | have confidence in the 
administrator’s ability to provide 
future leadership to the unit. 

e Strongly Disagree 

e Disagree 

e =In Between 

e §6Agree 

e §©Strongly Agree 


Although The IDEA Center discourages using these two items as the only source of evidence for 
summative decisions, respondent feedback about them can be helpful. One item deals with the 
administrator’s leadership up to the present time; the other expresses degree of confidence in the 
future. Such feedback can give a reading of the overall impressions of the administrator’s effectiveness. 


Open-ended Comments: 


The IA contained three prompts intended to obtain written comments about the administrator 
and the unit. In discussions with panel experts, minor modifications were made in the wordings of these 
items. The progression of changes in the items appears in Table 18. 


Table 18 


Progression of Wording in Open-ended Comments 


EASE Wording 


IA Wording 


IA-R Wording 


What are this administrator’s 
main strengths? 

What reservations do you 
have about this person as an 
administrator? 


31. What are this administrator’s 
main assets? 

32. What reservations do you have 
about this person as an 
administrator? 


28. What are this administrator’s 
main strengths? 

29. Do you have any reservations 
about this person’s ability to 
continue to serve in his or her 
current position? 


What changes (in priorities, 
style, organization, policy) do 
you feel would improve this 
administrator’s effectiveness? 


33. What changes (in priorities, style, 
organization, policy) do you feel 
would do most to improve this 
administrator’s effectiveness? 


30. What changes (in priorities, style, 
organization, policy) do you feel 
would do most to improve this 
administrator’s effectiveness? 
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Administrator Information Form: 


Initially, the items on the Administrator Information Form (AIF) were used for research purposes 
(see Appendix A). The AIF-R was expanded to include an option self-reflection where administrators 
provide ratings on the same items included in the IA-R (described in previous sections; see Appendix B). 
The resulting report provides a gap-analysis, showing where the administrator has a similar or a 
different self-perception than the raters (see Appendix E for a sample report). Table 19 shows the 
progress of wording for the items originally included in the AIF. 


Table 19 


Progression of Wording on Research Items included on the Administrator Information Form 


AIF Wording 


AIF-R Wording 


General Information 


1. Number of years at institution 


29. Including this year, number of years at this 
institution 


2. Number of years in this position at this institution 


30. Including this year, number of years in this position 
at this institution 


3. Number of years in this position at any institution 


31. Including this year, number of years in this position 
at any other institution 


Administrator Position 


4. When you first assumed this position, what was the 
general campus reputation of your unit in terms of 
importance/influence? 

e positive 

e neutral 

e ~=©negative 

e = =NA/new unit 
e = don’t know 


26. When you first assumed this position, what was the 
general campus reputation of your unit in terms of 
importance/influence? 

O positive 

O neutral 

© negative 

O not applicable/new unit 

0 don’t know 


5. When you first assumed this position, what was the 
general campus reputation of your unit in terms of 
effectiveness? 

e positive 

e neutral 

e = negative 

e =NA/new unit 
e = don’t know 


27. When you first assumed this position, what was the 
general campus reputation of your unit in terms of 
organizational effectiveness? 

O positive 

O neutral 

© negative 

O not applicable/new unit 

0 don’t know 


6. When you first assumed this position, you were 
expected to: 
e make major changes in the unit’s activities 
and/or the way they were conducted 
e primarily maintain established services and 
procedures 
e = other 


28. When you first assumed this position, you were 
expected to: 

oO make major changes in the unit’s activities and/or the 
way they were conducted 

oO primarily maintain established services and 
procedures 

o other 


7. Before you assumed this position, turnover of 
personnel in the unit had been: 
e ~— higher than most units 
e about average 
e lower than most units 
e = don’t know 
e = I’d prefer not to respond 


Table 18 continued on next page 
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Table 19 continued 


8. Since you have been in this position, personnel 
turnover has: 
e increased 
e remained constant 
e decreased 
e §=©don’t know 
e = I’d prefer not to respond 


9. Since you have been in this position, financial support 
for the unit, relative to that for other units at the 
institution, has: 

e increased 

e remained constant 

e decreased 

e = don’t know 

e = I’d prefer not to respond 


10. Since you have been in this position, the workload 
of the unit has: 
e increased 
e remained constant 
e decreased 
e §=©don’t know 
e = I’d prefer not to respond 


11. Including interim leadership, approximately how 32. Including interim leadership, approximately how 
many individuals, including yourself, have occupied many individuals, including yourself, have occupied 
your position during the past ten years? your position during the past 10 years? 

12. Number of people in this unit (omitting students) 33. Number of people in this unit (omitting students) 
classified as professionals? who report directly or indirectly to you? 

13. Number of people in this unit (omitting students) 34. Of those, how many people report directly to you? 


classified as support staff? 


Items 7-10 were not retained on the AIF-R. 


Summary: 

The wording of IA items underwent several transformations based on data analysis, feedback 
from clients, input from an expert panel, and review of relevant literature. Most of the items in the IA-R 
are tied to previous versions of administrator survey instruments used across the last two decades. Of 
the current 10 administrative roles, 7 came from similarly worded items on the EASE used at Kansas 
State University. Three new roles were added for the IA, which were then modified for the IA-R. Of the 5 
items that make up the Leadership Style scale, two were modified from the EASE and three were carried 
over from the IA, using slightly different wording. Three items on the EASE were retained for the 8-item 
Personal Characteristics scale, and five were modified from the IA. Two new characteristics were added 
to the IA-R: flexibility (Leadership Style subscale) and practical judgment (Personal Characteristics 
subscale). Slight modifications were made in the wordings of and scales associated with two summary 
judgment items. Likewise, two out of three open-ended comment items underwent minor changes. 


Responses made by the administrator on the AIF were used for research purposes (see 
Appendix A). The AIF-R retains modified versions of nine items and includes an optional self-reflection 
where the administrator rates him or herself on the same items included in the IA-R, with the exception 
of the overall impression items. 
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Process Modification 


Optional Gap Analysis: 

A key feature of the revised system is the provision of a gap analysis made possible by the 
administrator and personnel responding to the same survey items. Administrators rate themselves on 
the AIF-R regarding 10 administrative roles and 15 personal characteristics. Administrators can also add 
up to 20 questions to customize the instrument for specific roles. 


Personnel rate their administrator on the same 10 roles and 15 characteristics, using the IA-R, 
thereby enabling a gap analysis (both forms appear in Appendix B). Personnel also rate their confidence 
in the administrator’s ability to provide future leadership to the unit, and they offer their judgment of 
her/his past leadership. Finally, they respond in writing to three open-ended questions and indicate how 
they are affiliated with the institution (e.g., faculty member, student, etc.). 


Optional Sub-group Analysis: 

The IA form included a number of rater perspective items that allowed for results to be provided 
for atypical subgroups. However, after an analysis of that data and feedback from clients and experts, it 
was determined that those items were not always wanted as some clients felt they may make 
respondents question the confidentiality of their responses and could impact response rates. It was 
therefore decided to add one optional rater perspective item that would allow for subgroup results to 
be presented if at least 5 respondents belonged to a subgroup and no subgroup had only one 
respondent. Furthermore, the client can select from two to four response options to include. Table 20 
shows the item and possible response options. 


Table 20 

Optional Rater Perspective Item and Response Options 

Item: Client selects at least 2 and as many as 4 of the 
following options: 

What is the principal way that you are affiliated with e Faculty member 

this institution? OR 


o Adjunct faculty 
o Regular faculty 
e Student 
e Administrator's staff 
¢ Professional colleague/associate 
¢ Department head/chair 
e Dean 
¢ Governing board member 
¢ Community member 
e = Alumnus/alumna 
¢ Department head/chair 
e Dean 
e Other 


¢ | choose to omit my response to this question. (always 
included as an option) 
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Report Emphasis: 

The Administrator Report begins with a snapshot of the administrator’s overall effectiveness 
based on personnel perceptions of the administrator’s past leadership and his/her ability to provide 
leadership in the future. A summary of areas of strengths is then provided. Specific feedback is then 
given about the administrator’s performance of administrative roles, leadership style, and personal 
characteristics. If self-ratings were used, a gap analysis of the administrator’s ratings to the 
respondent’s ratings is included. Sub-group results are also provided in a graphical display (if used). 
Based on this information, the report identifies the administrator’s strengths and indicates areas for 
improvement, offering insights into how best to improve. The report concludes with detailed item 
statistics. Summarized feedback for additional questions and verbatim versions of personnel written 
responses to open-ended questions are provided in a separate report. 


Focus group participants and experts in the field provided a consistent message that it was more 
useful for administrators to consider perceptions of their staff than in how ratings compare to other 
administrators in different types of positions and institutions. Therefore, the IDEA Feedback for 
Administrators Report contains no normative information. However, once a sufficient number of 
administrators have participated in the revised system, normative information could be available for 
those interested in such interpretations. 


VALIDITY 


Validity refers to evidence that supports the proper use and interpretation of scores obtained 
from an assessment. The validity of any measure depends on how it is used in decision- making. The 
IDEA Feedback for Administrators instruments are primarily intended for formative and developmental 
purposes. The system is intended to measure (a) personnel perceptions of the administrator’s 
effectiveness and (b) the congruence between the administrator’s self-ratings and personnel ratings of 
his/her performance (i.e., gap analysis). The Administrator Report provides feedback about this 
information along with guidelines for interpreting that feedback and insights for improving 
administrative performance. If institutions choose to use the Administrator Report for summative 
purposes, the IDEA Center strongly recommends that additional indicators of effectiveness the 
institution deems appropriate also be considered. 


Evidence of Content Validity 


Content validity primarily refers to the wording and format of items in a survey. Evidence of 
content validity can also come from the judgments of experts (American Educational Research 
Association [AERA] et al., 1999). Two sources of expertise were considered in revising the AIF and 
personnel survey: administrators who participated in a focus group and expert panel members. Their 
input was used to identify items that were useful and appropriate for this instrument and to 
recommend changes in items, process, and reporting. Evidence for the content validity of items 
contained on the AIF-R and IA-R can be found in descriptions of how the questions were developed. 
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Validity Evidence Based on Internal Structure 


Evidence for the validity of an instrument can also come from its internal structure or observed 
relationships among items. One aspect of item-interrelationships concerns whether the underlying 
structure is comprised of a single dimension or is multi-dimensional. An assumption of the IDEA system 
is that the administrative unit’s personnel are good judges of the administrator’s performance. With this 
assumption in mind, we pursued the following investigations of the IDEA Administrator Feedback 
system’s internal structure. The following sections first summarize the analyses focusing on the 10 
Administrative Roles and then discuss the research focused on the 18 and Personal Characteristics 
items. 


Administrative Roles: 

To examine the internal structure of the Administrative Roles, correlations among the 10 roles 
were examined, a principal components analysis was conducted, and correlations with the two 
summary measures was examined 


Table 21 presents Pearson r correlations among personnel ratings of the administrative roles. All 
correlations were positive and strong (all rs > .72). The consistent pattern of high correlations suggests 
the items were measuring some common underlying dimension. (This possibility is further examined in a 
principal components analysis presented next). The following pairs of items had especially high 
correlations: 


e 2.Establishes sound priorities and 4. Makes wise decisions, judgments, and recommendations, r= 

.92 

3. Displays knowledge/expertise required for this position and 4. Makes wise decisions, judgments, 

and recommendations, r= .88 

e 6.ls an effective ‘team’ member and 7. Contributes positively to this institution’s image and 
reputation, r= .88 

e 9.Seeks opinions of others before establishing policies or procedures and 10. Earns the trust and 
respect of those who come in contact with him/her, r= .89 

e 7. Contributes positively to this institution’s image and reputation and 10. Earns the trust and 
respect of those who come in contact with him/her, r= .91 
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Table 21 
Pearson Product-Moment Correlations for Personnel Ratings of Administrative Roles (N =732) 


Responsibility 1 2 3 4 5 6 7 8 9 
1. Communicates and displays a visionary plan for 1 

the future. 

2. Establishes sound priorities. .86 1 


3. Displays knowledge/expertise required for this 
position. 

4. Makes wise decisions, judgments, and 
recommendations. 

5. Initiates actions that anticipate problems or 
resolves them. 


6. Is an effective "team" member. .79 .87 80 36.91 86 1 


7. Contributes positively to this institution’s image 
and reputation. 

8. Communicates relevant information to 
appropriate constituencies. 

9. Seeks opinions of others before establishing 
policies or procedures. 

10. Earns the trust and respect of those who come in 
contact with him/her. 


73 81 75 87 82 89 80 .88 1 


79 87 82 93 87 94 91 .87  .89 


A principal components analysis of the Administrator Roles was also conducted. The purpose of 
the following analyses was to identify the underlying factor structure of the personnel ratings of 
administrative roles. Prior to conducting the analyses the aggregated data file was split, assigning 50% of 
the cases to a derivation sample and 50% to a replication sample. Principal components analysis (PCA), 
with pairwise deletion, was performed on the derivation sample. This analysis yielded one clear factor 
(eigenvalue = 8.72), which explained 87.2% of the variance. The aggregated personnel ratings for each of 
the 10 roles had a component matrix coefficient of at least .88 and contributed substantially to the 
scale, which we named Administrative roles. The mean of the 10-item composite scale was 4.06 (SD = 
.58) on the five-point scale. The single factor was confirmed in the replication sample. These results 
provide construct validity evidence for a single dimension. Table 22 presents component matrix 
coefficients for personnel ratings of the entire sample of 732 administrators. 
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Table 22 
Component Matrix Coefficients for Personnel Performance Ratings of Administrative Roles (N = 732) 


Component 
Administrative Roles Matrix 

Coefficients 

4. Makes wise decisions, judgments, and recommendations. 97 

10. Earns the trust and respect of those who come in contact with 06 

him/her. 

6. Is an effective "team" member. 95 

2. Establishes sound priorities. 95 

5. Initiates actions that anticipate problems or resolves them. 94 

8. Communicates relevant information to appropriate constituencies. .93 

7. Contributes positively to this institution’s image and reputation. 92 

9. Seeks opinions of others before establishing policies or procedures. 91 

3. Displays knowledge/expertise required for this position. .90 

1. Communicates and displays a visionary plan for the future. .88 


Note: Extracted using principal components. 


Correlations between personnel ratings of Administrative Roles and Summary Judgments were 
also examined. It was hypothesized that personnel ratings of the administrator’s performance of 
administrative roles would be positively correlated with overall summary judgments. The coefficients 
presented in Table 23 confirm this. Personnel ratings of all 10 administrative roles showed high positive 
correlations with overall impressions of the administrator. The composite score on the 10-item 
Administrator Role scale was also highly correlated with Item 29 (r= .96) and Item 30 (r = .94) 


Table 23 
Pearson Product-Moment Correlations between Personnel Ratings of Administrative Roles and their 
Summary Judgments of the Administrator (N = 732) 


Responsibility neue? Hou ibs 
“Good Job” “Confidence” 
1. Communicates and displays a visionary plan for the future. .86 .79 
2. Establishes sound priorities. .92 .88 
3. Displays knowledge/expertise required for this position. .87 .86 
4. Makes wise decisions, judgments, and recommendations. .93 .92 
5. Initiates actions that anticipate problems or resolves them. 91 .88 
6. Is an effective "team" member. .90 91 
7. Contributes positively to this institution’s image and reputation. .88 .88 
8. Communicates relevant information to appropriate constituencies. .88 .87 
9. Seeks opinions of others before establishing policies or procedures. 85 .86 
10. Earns the trust and respect of those who come in contact with 
him/her. na a 


Note: Higher values for Items 29 and 30 reflect more positive evaluations. 
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Personal Characteristics: 

To examine the internal structure of the Personal Characteristics, correlations among the 18 
items were examined, a principal components analysis was conducted, and correlations with the two 
summary measures was examined 


Table 24 presents Pearson r correlations among ratings of the administrator’s personal 
characteristics. Because the direction of the scale varied, the positively worded items (1 = positive 
characteristic) were re-coded so that higher values for every personal characteristic represented more 
positive evaluations of the administrator. High correlations between items assessing similar constructs 
(e.g., Remote vs. Approachable, Unfair vs. Fair, Autocratic vs. Democratic, Unfeeling vs. Caring, 
Opinionated vs. Receptive to ideas, Aloof vs. Warm) suggested the possibility of a common underlying 
dimension. Low correlations indicated different constructs were being measured (e.g., Indecisive vs. 
Decisive and Autocratic vs. Democratic; Indecisive vs. Decisive and Opinionated vs. Receptive to ideas). 
(We explored these possibilities, reported in the pages that follow). The following high correlations 
provided evidence of considerable redundancy among items: 


e 14. Untruthful vs., Honest and 25. Untrustworthy vs. Trustworthy, r= .86 

e 13. Remote vs. Approachable and 27. Aloof vs. Warm, r= .87 

e 15. Unfair vs. Fair and 16. Autocratic vs. Democratic, r= .83 

e 18. Manipulative vs. Straightforward and 25. Untrustworthy vs. Trustworthy, r= .87 
e 19. Inconsistent vs. Consistent and 28. Erratic vs. Predictable, r= .82 

e 20. Lethargic vs. Vigorous and 26.Passive vs. Active, r= .86 

e 17. Unfeeling vs. Caring and 23. Insensitive vs. Understanding, r= .90 
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Table 24 
Pearson Product-Moment Correlations for Personnel Ratings of Administrators’ Personal Characteristics (N = 732) 


Responsibility 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 
11. Indecisive vs. Decisive 1 

12. Disorganized vs. Organized 73 1 

13. Remote vs. Approachable 35 33 1 

14. Untruthful vs. Honest 55 55 57 1 

15. Unfair vs. Fair 47 47 77 78 1 

16. Autocratic vs. Democratic .27 34 78 .68 .83 1 

17. Unfeeling vs. Caring 43 40 76 77 79 .80 1 

18. Manipulative vs. Straightforward 49 .48 .67 82 85 77 74 1 

19. Inconsistent vs. Consistent .68 72 54 75 71 61 .67 71 1 

20. Lethargic vs. Vigorous .62 .50 48 43 49 37 43 47 49 1 

21. Ambiguous vs. Clear .69 .69 61 71 76 .64 .66 77 82 .63 1 

22. Self- vs. Institution-centered 49 49 .66 73 77 .69 71 .80 64 59 71 1 

23. Insensitive vs. Understanding 44 43 .80 77 .86 85 .90 79 .69 45 .70 74 1 

acne YOORPERPENVE FO 35 441 (75 69 “80 87 99 395 “66 40 66 71 87. 4 

25. Untrustworthy vs. Trustworthy 53 52 .67 .86 .83 72 79 .87 76 48 .76 78 .84 75 1 

26. Passive vs. Active .70 52 45 42 46 30 41 43 50 .86 .63 54 41 34 45 1 

27. Aloof vs. Warm 36 32 .87 .62 .80 81 84 70 58 52 65 72 85 79 72 47 1 

28. Erratic vs. Predictable 54 .64 50 .68 .67 .62 .60 .68 82 38 75 .64 .66 .66 72 35 54 
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The underlying factor structure of personnel ratings of the administrator’s Personal 
Characteristics was also examined using a principal components analysis (PCA). Again, the aggregated 
data file was split. PCA initially yielded three factors having eigenvalues greater than 1.0, which in 
combination explained a total of 83.2% of the variance in the derivation sample. The Scree plot showed 
a large drop off in eigenvalues from the first to the second factor, and then less noticeable declines from 
the second to the third factors. At that point and beyond a leveling off could be observed. In 
determining the final number of factors to extract, we applied two criteria: (1) a rotated pattern matrix 
coefficient of at least .40, and (2) a minimum of two items per factor. Following varimax rotation, the 
first factor explained 40.4% of the variance (eigenvalue = 7.27), the second 27.7% (eigenvalue = 4.99), 
and the third 15.1% (eigenvalue = 2.72). The first factor, which was comprised of 9 items, appeared to 
represent Personal Characteristics. The second factor was made up of 6 items and seemed to 
characterize Leadership Style. The third factor was associated with 2 items concerned with Vigor. In the 
replication sample, the three-factor structure was confirmed. However, the patterns of loadings 
changed: The first factor contained 11 items and the second factor 4 items. 


Following those analyses, we re-examined the high correlations and conceptual redundancy 
among the following pairs of items: 


e Untrustworthy vs. Trustworthy and Untruthful vs. Honest 
e Remote vs. Approachable and Aloof vs. Warm 

e Inconsistent vs. Consistent and Erratic vs. Predictable 

e Passive vs. Active and Lethargic vs. Vigorous 

e Insensitive vs. Understanding and Unfeeling vs. Caring 


We then removed the items above in bold from the analysis and conducted PCA on the 
remaining 13 items within the derivation sample. Only two factors had eigenvalues greater than 1. The 
first factor explained 69.3% of the variance (eigenvalue = 9.01), and the second 11.1% (eigenvalue = 
1.56). Following varimax rotation, the first factor explained 47.1% of the variance (eigenvalue = 6.12) 
and was associated with 8 items having rotated component matrix coefficients > .40. Those 8 items had 
previously loaded on the Personal Characteristics factor, and so that factor name was retained. All items 
had high item-total correlations with the overall scale score. The mean composite score for items 
loading on the 8-item factor was 5.60 (SD = .73) on the seven-point scale. The second rotated factor, 
which explained 34.2% of the variance (eigenvalue = 4.44), was comprised of five items with coefficients 
exceeding .40. Because those five items had previously loaded on the Leadership Style factor, that factor 
name was also retained. All five items had high item-total correlations with the overall scale score. The 
mean composite score for the five-item scale was 5.59 (SD = .61). This factor structure was confirmed in 
the replication sample. Component matrix coefficients for the entire sample of personnel ratings of 
administrators may be found in Table 25. 


As a check to make sure the pattern of correlations among the 18 personal characteristics items 
was consistent among subgroups, we conducted PCA on subgroups defined in Items 35 and 36. The 
pattern of correlations and component matrix coefficients were highly consistent among subgroups. We 
also attempted to force a three-factor solution with the five items deleted; however, the third factor 
was associated with only one item (Passive vs. Active). 


32 


Table 25 
Rotated Component Matrix Coefficients for Personnel Ratings of Personal Characteristics with Five Items 
Deleted (N = 732) 


Factors 
Characteristics Reichl: Penee rsp 

Characteristics Style 
16. Autocratic vs. Democratic 0.93 0.12 
23. Insensitive vs. Understanding 0.90 0.28 
24. Opinionated vs. Receptive to ideas 0.89 0.21 
15. Unfair vs. Fair 0.87 0.35 
13. Remote vs. Approachable 0.83 0.20 
18. Manipulative vs. Straightforward 0.81 0.40 
25. Untrustworthy vs. Trustworthy 0.79 0.44 
22. Self-centered vs. Institution-centered 0.74 0.44 
11. Indecisive vs. Decisive 0.17 0.91 
12. Disorganized vs. Organized 0.22 0.84 
26. Passive vs. Active 0.21 0.77 
21. Ambiguous vs. Clear 0.59 0.70 
19. Inconsistent vs. Consistent 0.56 0.68 


Correlations between personnel ratings of Personal Characteristics and Summary Judgments 
were also examined. It was hypothesized that positive correlations would exist between respondent 
ratings on all 18 personal characteristics and overall summary judgments of the administrator (Items 29 
and 30). Table 26 presents those Pearson product-moment correlations. Most correlations were 
moderate-to-high in magnitude. In Table 27, the same pattern is indicated for the relationships between 
the two summary judgment items and the composite scale scores for Personal Characteristics and 
Leadership Style. 
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Table 26 
Pearson Product-Moment Correlations between Personnel Ratings of 
Personal Characteristics and Summary Judgments of the Administrator (N = 732) 


Characteristic Heme? em 20 
“Good Job” “Confidence” 
11. Indecisive vs. Decisive .67 .60 
12. Disorganized vs. Organized .63 57 
13. Remote vs. Approachable .69 .69 
14. Untruthful vs. Honest 72 76 
15. Unfair vs. Fair 81 .84 
16. Autocratic vs. Democratic 72 75 
17. Unfeeling vs. Caring 73 74 
18. Manipulative vs. Straightforward 79 .83 
19. Inconsistent vs. Consistent 18 75 
20. Lethargic vs. Vigorous .67 .61 
21. Ambiguous vs. Clear 85 .82 
22. Self-centered vs. Institution-centered 78 .80 
23. Insensitive vs. Understanding 78 .80 
24. Opinionated vs. Receptive to ideas 72 75 
25. Untrustworthy vs. Trustworthy .79 .84 
26. Passive vs. Active .67 .61 
27. Aloof vs. Warm .73 73 
28. Erratic vs. Predictable .70 .69 


Note: Higher values for Items 29 and 30 reflect more positive evaluations. 


Table 27 
Pearson Product-Moment Correlations between Personnel Ratings of Factor Scale Scores and 
Summary Judgments of the Administrator (N =732) 


Bahan ceaia Item 29 Item 30 
“Good Job” “Confidence” 

2. Interpersonal Characteristics (8-item scale) 83 .86 

3. Leadership Style (5-item scale) 85 .79 


Note: Higher values for Items 29 and 30 reflect more positive evaluations. 


Summary: 

PCA revealed that clusters of IA items were most likely measuring the same underlying 
dimension or factor (i.e., latent variable). Personnel ratings of 10 “strengths and weaknesses” clustered 
around a single dimension, which we named Administrative Roles. Responses to 8 items were associated 
with the latent trait of Personal Characteristics, and 5 others measured Leadership Style. Moreover, the 
pattern of correlations and component matrix coefficients was consistent among subgroups of rater 
characteristics, which again provides evidence of validity generalization. Personnel ratings on all IA items 
and subscales were highly correlated with two summary judgments, which provides additional evidence 
of validity based on internal structure. 
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Validity Evidence Based on Relationships to Other Variables 


Relationships of items and scales to variables external to the instrument can provide another 
important source of validity evidence. Demographic variables, for example, which include group 
memberships, are relevant when it is expected that group differences should be present or absent if 
interpretations of the instrument are to be supported (AERA, APA, NCME, 1999). In this section, we 
present evidence of when group differences are absent—which supports consistency in interpretations 
across demographic variables—and evidence of when group differences are present. 


Comparison of Factor Composite Scale Scores by Demographics of the Administrator and Perceptions 
of the Unit: 

We computed mean scores from summed aggregated ratings on Administrative Roles (10 
items), Interpersonal Characteristics (8 items), and Leadership Style (5 items). Pearson r correlations 
between each factor scale and the administrator’s years at the institution, years in the position, years in 
the position at any institution, number asked to respond, number of professionals in the unit, number of 
support staff in the unit, and response rate are presented in Table 28. For the most part, those 
correlations indicated negligible relationships existed between the factors and administrator 
demographics. We also conducted ANOVAs to investigate whether mean scores on the scales varied by 
the administrator’s perceptions of the characteristics of the unit (i.e., Items 4 to 10, Table 5). For Items 
4,5, and 7 to 10, respondents choosing “NA/new unit,” “don’t know,” or “I’d prefer not to respond,” 
were not included in the analyses. Those selecting “other” in Item 6 also were also not included. We 
tested all main effects and any meaningful two-way interactions that produced large enough cell sizes 
(i.e., Item 4 with Item 5, Item 7 with Item 8). Because of the number of analyses performed, Type | error 
rate was set at a= .01. We considered effect sizes less than .01 to be of little practical significance. The 
following meaningful effects emerged: 


1. Administrators received higher ratings on Administrative roles (1 to 5 scale) when—at the time 
of appointment—they perceived the general campus reputation of the unit, in terms of 
effectiveness, to be positive (MV = 4.15, SD = .50) or neutral (M = 4.15, SD = .47) rather than 
negative (M = 3.92, SD = .54), F (2,573) = 12.79, p< .001, He = 043. 

2. Administrators received higher ratings on Interpersonal Characteristics (1 to 7 scale) when—at 
the time of appointment—they perceived the general campus reputation of the unit, in terms of 
effectiveness, to be positive (MV = 5.76, SD = .58) or neutral (M = 5.74, SD = .66) rather than 
negative (M = 5.38, SD =.71), F (2,573) = 20.25, p< .001, Np = 066. 

3. Administrators received higher ratings on Administrative roles, when—at the time of 
appointment—they perceived that they were expected to primarily maintain “the status quo” 
(M = 4.20, SD = .46) than to make major changes (M = 4.04, SD = .53), F (1,633) = 11.97, p< .001, 
Np = 019. 

4. Administrators received higher ratings on Interpersonal Characteristics, when—at the time of 
appointment—they perceived that they were expected to primarily maintain “the status quo” 
(M = 5.86, SD = .54) than to make major changes (M = 5.56, SD = .69), F (1,633) = 26.65, p< .001, 
fp = 104, 


In general, then, personnel gave the highest ratings to administrators who believed they were 


expected to maintain the status quo and who perceived their units as already having good reputations 
at the time of their appointment. This suggests that perceptions of the unit and expectations placed on 
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the administrator are important extraneous factors to consider when interpreting findings from the 
Administrator Report. 


Table 28 
Pearson Product-Moment Correlations between Factor Scale Scores and Administrator Demographics (N 
= 732) 


Variable 1 2 3 4 5 6 7 8 9 
1. Administrative Roles 1 

2. Interpersonal Characteristics .89 1 

3. Leadership Style 83 69 1 

4. Years at institution 01 -.05 .04 1 

5. Years in position 03 -.03 -.02  .45 1 

6. Years in position at any institution .00 -.04 -.05  .18 .63 1 

7. Number asked to respond -16 -.16 -11 .06 -.06 -.07 1 

8. Response rate .36 35 25 -.08 .05 11 «-.45 1 
oar oliproressiqnals (not oa Sod: 203 307 <06 “2:03 200. <o7) -a 
ae eee ot suppor stalhingt 02-04 -01 .07 01 .00 02 .03 .56 


Comparisons of Summary Judgments by Demographics of the Administrator and the Administrator’s 
Perceptions of the Unit: 

Analyses were conducted to examine whether respondents’ summary judgments (IA Items 29 
and 30) of the administrator varied as a function of the administrator’s demographic characteristics and 
perceptions of the unit. ANOVAs were conducted on two summary judgments (dependent variables) by 
the administrator’s rating of (a) the unit’s general reputation of importance/influence and effectiveness 
at the time the administrator was appointed, expectations for the administrator upon his/her initial 
employment, turnover in the unit, financial support, and workload (independent variables). We also 
correlated the two summary judgments with the administrator’s years in the position, the institution, 
and any institution; the number of professionals and support staff in the unit; and how many individuals 
had occupied the position in the previous 10 years. To examine possible moderating effects, all possible 
combinations of two-way interactions were computed for the aforementioned rater characteristics. 
Because of the number of statistical tests conducted, Type | error rate was set at .01 for all analyses. No 
interactions were significant at the @ = .01 level on either of the summary judgments. 


For two AIF items, main effects were found on the summary judgment items. First, AIF Item 5 
asked “When you first assumed this position, what was the general campus reputation of the unit in 
terms of effectiveness?” ANOVAs were conducted on only those responding “positive,” “neutral,” or 
“negative.” (Those who responded “NA/new unit” or “don’t know” were left out of the analysis). Results 
indicated that administrators’ responses to Item 5 were related to respondents overall impressions of 
the kind of job the administrator was doing, F (2,584) = 10.86, p < .001, Np = .036; and their overall 
confidence, F (2, 584) = 14.57, p < .001, Np? = .048. Table 29 presents descriptive statistics and the 
results of post-hoc tests for both dependent variables. The personnel had higher overall impressions 
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when, at the time of appointment, the administrator perceived the unit’s effectiveness as either positive 
or neutral than if it were perceived as negative. 


Table 29 
Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary Judgments by 
the Administrator’s Perception of the Unit’s Reputation of Effectiveness 


When you first assumed this position, what was the general campus 


reputation of your unit in terms of effectiveness? 


. Positive Neutral Negative 
Overall Impression Item 
M SD n M SD n M SD n 
29. What kind of job is this administrator doing? 3.78, 055 201 | 3.82, 0.55 195 | 3.57, 0.58 191 
30. Does this administrator have your confidence? 3.57, 0.40 201 | 3.58, 0.38 195] 3.39, 0.43 191 


Note. Means with different subscripts within rows differ significantly (Tukey HSD, a< .01). 
SD = Standard deviation. 
Personnel responded on a 1 (Poor) to 5 (Superb) scale for Item 29. 
Personnel responded on a 1 (Definitely Not) to 4 (Definitely Yes) scale for Item 30. 


Second, the overall ratings of confidence (Item 30), t (408.66) = 4.11, p < .001, Mp = .022 were 
higher when the administrator perceived he/she was expected to primarily maintain established 
services and procedures than to make major changes (see Table 30); however, results for Item 30 had 
little practical significance.*? Nonetheless, the findings indicate that leadership may be more difficult 
when change is mandated. These were the only administrator and unit characteristics significantly 
related to overall summary judgments. 


Table 30 
Means Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary Judgments by 
the Administrator’s Expectations 


When you first assumed this position, 
you were expected to: 


Maintain established services 


Make major changes and procedures 


Overall Impression Item 


M SD n M SD n 
29. What kind of job is this administrator doing? 3.71 0.59 454 3.83 0.53 181 
30. Does this administrator have your confidence? 3.49, 0.43 454 3.62, 0.35 181 


Note. Means with different subscripts within rows significantly differ (Tukey HSD, a< .01). 
SD = Standard deviation. 
Personnel responded on a 1 (Poor) to 5 (Superb) scale for Item 29. 
Personnel responded on a 1 (Definitely Note) to 4 (Definitely Yes) scale for Item 30. 


Comparisons of Personnel Ratings of Administrative roles and Personal Characteristics by Rater 
Perspectives: 

Possible differences in personnel ratings of the 10 administrative roles and 18 personal 
characteristics based upon rater perspectives found in Table 6 were investigated. ANOVAs were 
conducted on each of the 10 roles and 18 personal characteristics (dependent variables) by several rater 


> For Item 30, degrees of freedom based on t-test using separate variance estimate. 
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perspectives (independent variables): affiliation with the institution, principal type of contact with the 
administrator, frequency of meaningful contact with the administrator, and length of employment at 

the institution. Given the enormous power of the analyses (over 27,000 personnel responding), trivial 
effects are more likely to be significant. We therefore set Type | error rate at @ = .01 and reported 


effect sizes (75) for each significant effect. Post-hoc analyses were then conducted for any significant 


rater effects with more than two subgroups. 


The only meaningful difference was observed on Item 36, “How often do you have meaningful 
contact with the administrator?” Table 31 presents means and standard deviations for personnel ratings 
of Administrative Roles at each level of meaningful contact; Table 32 presents the same information for 
ratings of Personal Characteristics. Across all 10 administrative roles, meaningful contact with the 
administrator consistently accounted for at least or more than 2% of the variance (7; > .02). 
Meaningful contact with the administrator also explained 2% of the variance for ratings on 12 of the 18 
personal characteristics. Specifically, personnel provided more positive evaluations in a linearly 
increasing fashion, from those hardly ever having meaningful contact with their administrator to those 
having daily contact. The more frequently raters had contact with their administrator, the more likely 
they were to rate her or him positively on all characteristics. 
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Table 31 


Descriptive Statistics and ANOVA Results for Ratings of 10 Administrative roles by Frequency of Meaningful Contact with the Administrator 


(disaggregated data) 


How often do you have meaningful contact with this administrator? 


Daily At least once /week 1-2 times/month 1-2 times/term Less than once/term | Omnibus Test 
M sd N | mM so wn |mM so. WN |M SD Nw |M SDN p Ny 
1. Communicates and displays a visionary, ,5 993 4963| 4.03, 1.06 4,153|3.96, 1.10 5,055 | 3.94, 1.12 5,039) 3.67, 124 3,953 | <001 024 
plan for the future. 
2. Establishes sound priorities. 4.36, 0.89 1,976| 4.09, 1.00 4,132 | 4.00, 1.06 4,918] 3.93, 1.07 4,802 | 3.62, 1.17 3,611 | <.001 .036 
3. Displays knowledge/expertise required 4 6, 977 1981 | 4.40, 0.92 4,245 | 4.31, 0.99 5,269 | 4.23, 1.02 5,359/3.92. 114 4,049 | <001 040 
for this position. 
4. Makes wise decisions, judgments, and 
4.42, 0.87 1,977] 4.14, 1.02 4,209 | 4.03, 1.07 5,153] 3.94, 1.12 5,055 | 3.58, 1.21 3,650 | <.001 .045 
recommendations. 
5. Initiates actions that anticipate 
4.24, 0.98 1,963] 3.99, 1.09 4,119 | 3.87, 1.14 4,824] 3.79, 1.17 4,546 | 3.41, 1.28 3,101 | <.001 .039 
problems or resolves them. 
6. Is an effective "team" member. 4.47, 0.93 1,979] 4.21, 1.08 4,190 | 4.11, 1.14 5,111] 4.02, 1.17 4,857 | 3.67. 1.28 3,306 | <.001 .036 
7. Contribut itively to thi 
Si aa atl pees abe fone ay 4.66, 0.76 1,992 | 4.44, 0.93 4,229| 4.35, 1.00 5,275 | 4.28, 1.05 5,356 | 4.03, 1.12 4,280 | <.001 .048 
institution’s image and reputation. 
8. Communicates relevant informationto 435 493 1955]4.16, 1.06 4,145 | 4.07, 1.10 5,072| 4.00, 1.13 5,030]3.61, 1.25 3,675 | <.001 .037 
appropriate constituencies. 
Pace ks Opinions of omnets. Betore 4.33, 0.97 1,928 | 4.06, 1.13 4,068 | 3.89, 1.21 4,799| 3.74, 1.27 4,409 | 3.29, 1.38 2,925 | <.001 057 
establishing policies or procedures. 
10. Earns the trust and respect ofthose 4 as 95 4.975] 4.19, 1.12 4,232] 4.10, 1.16 5,248| 4.04, 1.17 5,280|3.72, 128 3,793 | <.001 .029 


who come in contact with him/her. 


Note. 


Means within same row are significantly different (@ = .05) if they have different subscripts, using the Tukey HSD post-hoc test. The p-values indicate 


significance levels for omnibus F-test comparing differences between categories of meaningful contact. The partial 7 indicates the proportion of 
variance that meaningful contact with the administrator explains in respondent ratings of each behavior. 
SD = Standard deviation. Personnel responded to all items on a 1 (Definite Weakness) to 5 (Definite Strength) scale. 
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Table 32 
Descriptive Statistics and ANOVA Results for Ratings of 18 Personal Characteristics by Meaningful Contact with the Administrator (disaggregated 
data) 


How often do you have meaningful contact with this administrator? 


Daily At least once /week 1-3 times/month 1-3 times/term Less than once/term Omnibus Test 

M SD N M SD N M SD N M SD N M SD N p Np 

11. Indecisive vs. Decisive 6.01, 1.49 1,965 | 5.82, 1.54 4,178 | 5.77, 1.55 5,106 | 5.74, 1.60 4,868 | 5.44. 1.80 3,100 | <.001 .011 
12. Disorganized vs. Organized 5.93, 1.55 1,985 | 5.85,, 1.57 4,164 | 5.87,, 1.52 5,053 | 5.81, 1.60 4,838 | 5.56, 1.76 2,950 | <.001 .006 
13. Remote vs. Approachable 5.90, 1.72 1,983 | 5.63, 1.81 4,265 | 5.48, 1.87 5,342 | 5.34, 1.92 5,443 | 4.49, 2.11 4,002 < .001 .05 
14. Untruthful vs. Honest 6.31, 1.60 1,973 | 6.15, 1.62 4,188 | 6.08, 1.67 5,145 | 5.91, 1.79 4,970 | 5.50, 1.98 3,080 | <.001 .022 
15. Unfair vs. Fair 6.16, 1.42 1,974] 5.92, 1.51 4,171 | 5.77, 1.64 5,069 | 5.66, 1.68 4,813 | 5.12, 1.83 2,972 < .001 .032 
16. Autocratic vs. Democratic 5.63, 1.64 1,885 | 5.14, 1.80 4,033 | 5.00, 1.87 4,873 | 4.86, 1.92 4,587 | 4.24, 2.02 2,930 | <.001 .038 
17. Unfeeling vs. Caring 6.20, 1.50 1,985 | 5.98, 1.59 4,226 | 5.84, 1.65 5,181 | 5.70, 1.74 5,124 | 5.16, 1.98 3,375 < .001 .033 
18. Manipulative vs. Straightforward 6.14, 1.36 1,972 | 5.75, 1.62 4,178 | 5.70, 1.65 5,137 | 5.61, 1.68 4,934 | 5.14, 182 2,988 | <.001 .025 
19. Inconsistent vs. Consistent 5.98, 1.62 1,968 | 5.81, 1.67 4,174 | 5.79, 1.65 5,057 | 5.69, 1.74 4,815 | 5.31, 1.92 2,996 | <.001 .015 
20. Lethargic vs. Vigorous 6.07, 1.28 1,933 | 5.90, 1.33 4,155 | 5.83, 1.33 5,171 | 5.74, 1.39 5,115 | 540, 159 3,460 | <.001 .019 
21. Ambiguous vs. Clear 5.80, 1.47 1,979 | 5.56, 1.55 4,229 | 5.49,. 1.60 5,255 | 5.45, 1.62 5,227 | 5.074 1.76 3,489 | <.001 .015 
22. Self-Centered vs. Institution-centered 6.23, 1.31 1,962 | 5.95, 1.50 4,196 | 5.85, 1.57 5,193 | 5.79, 1.58 5,101 | 5.49, 1.73 3,503 < .001 .016 
23. Insensitive vs. Understanding 6.31, 1.31 1,986 | 6.01, 1.45 4,230 | 5.87, 1.55 5,212 | 5.75, 1.62 5,120 | 5.18, 1.93 3,214) <.001 .040 
24. Opinionated vs. Receptive to ideas 6.02, 1.55 1,978 | 5.70, 1.74 4,223 | 5.57, 1.80 5,199 | 5.50, 1.84 5,105 | 4.93, 2.04 3,226 | <.001 .028 
25. Untrustworthy vs. Trustworthy 6.44, 1.34 1,973 | 6.24, 1.44 4,171 | 6.11, 1.53 5,117 | 5.96, 1.63 4,937 | 5.51, 1.89 3,117 | <.001 .033 
26. Passive vs. Active 6.04, 1.37 1,967 | 5.86, 1.46 4,204 | 5.76, 1.49 5,205 | 5.69, 1.50 5,179 | 5.31g 1.73 3,545 < .001 .019 
27. Aloof vs. Warm 5.89, 1.43 1,967 | 5.60, 1.57 4,226 | 5.48, 1.63 5,278 | 5.35, 1.68 5,318 | 4.73, 1.93 3,736 | <.001 .039 
28. Erratic vs. Predictable 5.90, 1.44 1,935 | 5.76, 1.46 4,076 | 5.75, 1.40 4,951 | 5.73, 144 4,592 | 5.44. 163 2,767 | <.001 .010 


Note. Means within same row are significantly different (@ = .05) if they have different subscripts, using the Tukey HSD post-hoc test. The p-values indicate 
significance levels for omnibus F-test comparing differences between categories of meaningful contact. The partial 7 indicates the proportion of 
variance that meaningful contact with the administrator explains in respondent ratings of each personal characteristic. 

SD = Standard deviation. Personnel responded on a 1 to 7 semantic differential scale. 
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Comparisons of Summary Judgments by Rater Perspectives: 

We also examined whether respondents’ summary judgments varied by the rater’s perspectives. 
ANOVAs were computed on disaggregated personnel ratings to determine whether the two summary 
judgments (dependent variables) varied by the respondent’s position, principal type of contact with the 
administrator, frequency of contact with the administrator, and length of service at the institution 
(independent variables). We conducted linear mixed-model ANOVAs, using administrator ID as a 
random variable. The general pattern of results was the same regardless of the ANOVA procedure 
employed. Because of the number of statistical tests conducted, Type | error rate was set at .01 for all 
analyses. 


Table 33 presents descriptive statistics and post-hoc tests for the results of the ANOVAs 
performed on overall summary judgments by respondents’ principal affiliation with the institution (Item 
34). The largest effect due to position was found on Item 29, “What kind of job is this administrator 
doing?” [F(3,21159) = 227.24, p< .001, Np = .031]. On the five-point scale, faculty rated administrators 
approximately one-half point lower than did students and .36 lower than did administrative staff and 
associates. There was also a significant effect on overall confidence in the administrator (Item 30), 
F(3,20866) = 219.94, p< .001, Np = .031. Faculty reported lower confidence compared to students, 
administrative staff, and associates. Students rated administrators higher than all others on both items. 


Table 33 
Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary Judgments by 
Respondents’ Position 


What is the principal way you are affiliated with this institution? 


. Faculty Member Student Administrative Staff | Meaningful Contacts 

Overall Impression 
M SD n M SD n M SD n M SD n 

29. What kind of job is 
this administrator 3.46, 1.14 11,263 | 4.03, 0.93 591 | 3.82, 1.01 4,233 | 3.82, 0.94 5,076 
doing? 
30. Does this 
administrator have 3.31, 0.98 11,054 | 3.67, 0.70 576 |] 3.57. 0.79 4,153 | 3.64,. 0.70 5,087 
your confidence? 


Note. Means with different subscripts within rows significantly differ (Tukey HSD, a= .01). 
SD = Standard deviation. 
Personnel responded on a 1 (Poor) to 5 (Superb) scale for Item 29. 
Personnel responded on a 1 (Definitely Note) to 4 (Definitely Yes) scale for Item 30. 


Table 34 presents the results of respondents’ summary judgments by their principal type of 
contact with the administrator (Item 35). First, there was a significant effect of type of contact on 
respondents’ impressions of the kind of job the administrator was doing (Item 29), F(3,19053) = 160.71, 
p<.001, Np = .025. Colleagues gave the highest ratings, followed by subordinates, both of whom gave 
higher ratings than did clients and those with informal relationships. A smaller significant effect was 
found for the rater’s overall confidence in the administrator (Item 30), F(3,18842) = 81.25, p < .001, ip = 
.013; colleagues reported higher confidence than all others. Subordinates also reported more 
confidence than did clients and those with informal relationships. 
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Table 34 
Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary Judgments by 
Respondents’ Principal Type of Contact with the Administrator 


What is the principal type of contact you have with this administrator? 


. Colleague Subordinate Client Informal 

Overall Impression 

M SD n M SD n M SD n M SD n 
29. What kind of job is 
this administrator 3.96, 0.96 3,780 | 3.79, 1.08 5,697 | 3.57, 1.08 4,647 | 3.54, 0.97 4,933 
doing? 
30. Does this 
administrator have 3.69, 0.69 3,783 | 3.53, 0.84 5,630 | 3.43, 0.90 4,596 | 3.44, 0.83 4,837 
your confidence? 


Note. Means with different subscripts within rows significantly differ (Tukey HSD, a= .01). SD = Standard 
deviation. 
Personnel responded on a 1 (Poor)-to-5 (Superb) scale for Item 29. 
Personnel responded on a 1 (Definitely Note)-to-4 (Definitely Yes) scale for Item 30. 


Table 35 presents results for overall impressions by the frequency of meaningful contact the 
respondent had with the administrator (Item 36). This variable explained 6.3% of the variance in 
respondents’ ratings of the kind of job the administrator was doing (Item 29), F(4,20770) = 348.86, p< 
.001, Np = .063. Figure 2 shows the linear relationship between the frequency of contact and overall 
impressions of the administrator. Respondents who reported daily contact with the administrator rated 
him/her, on average, almost a full point higher than did those whose contact was less than once per 
term. The effect of this rater perspective on overall confidence in the administrator (Item 30) was 
slightly smaller, F(4,20496) = 228.37, p < .001, Np = .043; but it followed the same trend. 


Table 35 
Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Summary Judgments by 
Respondents’ Meaningful Contact with the Administrator 


Meaningful Contact with Administrator 


Overall Daily At least once/week 1-3 times/month 1-3 times/term Less than once/term 
Impression M SD n M SD n M SD n M SD n M SD n 


29. What kind 
of job is this 
administrator 
doing? 

30. Does this 
administrator 
have your 
confidence? 


4.15, 0.89 1,986) 3.87, 1.01 4,236} 3.72, 1,04 5,222] 3.60, 1.03 5,299} 3.23, 1.04 4,032 


3.78, 0.58 1,967/ 3.60, 0.76 4,207] 3.53, 0.82 5,200} 3.46, 0.86 5,208) 3.17, 0.99 3,919 


Note. Means with different subscripts within rows significantly differ (Tukey HSD, a= .01). 
SD = Standard deviation. 
Personnel responded on a 1 (Poor)-to-5 (Superb) scale for Item 29. 
Personnel responded on a 1 (Definitely Note)-to-4 (Definitely Yes) scale for Item 30. 


42 


Figure 2 


Personnel Ratings of What Kind of Job the Administrator is doing by Frequency of 
Meaningful Contact with the Administrator 


What kind of a job is this administrator doing? 


4.2 


3.8 


3.6 


Estimated Marginal Means 


3.4 


3.2 


At least once a 
week, but not 
daily. 


On a daily basis. 


One to three times One to three times Less than oncea 
a month. aterm. term. 


How often do you have meaningful contact with this administrator? 


Table 36 presents the results of raters’ overall impressions by how long they had been at the 
institution (Item 37). Length of service at the institution had a declining linear effect on ratings of overall 
impressions. Those serving more than 5 years rated the administrator about a half point lower on Item 
29 than did those serving in their first year, F(3,20745) = 82.85, p < .001, Np = .012. The same trend was 
observed on Item 30, F(3,20420) = 83.70, p < .001, Np = .012. 


Table 36 


Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Overall Impressions by 
Respondents’ Length of Time at the Institution 


Respondent Length of Time at Institution 


. First Year 1-2 years 3-5 years More than 5 years 

Overall Impression 
M SD n M SD n M SD n M SD n 

29. What kind of job is 
this administrator 3.98, 0.84 978 | 3.85, 0.97 1,719 | 3.73, 1.02 3,726 | 3.574 1.11 14,326 
doing? 
30. Does this 
administrator have 3.75, 0.56 935 | 3.62, 0.74 1690 | 3.53c 0.81 3636 | 3.40, .93 14,163 
your confidence? 


Note. Means with different subscripts within rows significantly differ (Tukey HSD, @=.01). 


SD = Standard deviation. 
Personnel responded on a 1 (Poor) to 


5 (Superb) scale for Item 29. 


Personnel responded on a 1 (Definitely Note) to 4 (Definitely Yes) scale for Item 30. 
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Differences in Ratings of Overall Impressions between Raters Responding to, and those omitting 


Responses to, Rater Perspective Questions: 


As indicated in Table 6, a fairly high percentage of individuals chose not to respond to questions 
about the rater’s perspective (Items 34 to 37). Non-respondents ranged from 14% to 25% per item. 
When the number of non-respondents (i.e., missing values) totaled across Items 34 to 37 (M =.70, SD = 
1.19, MIN = 0, MAX = 4) was correlated with the overall response rate to the survey, the relationship 
was slightly negative, r = -.11. Thus, administrators with low overall response rates to the survey had 


more non-responses to rater perspective questions. 


To examine whether non-respondents to Items 34 to 37 provided different overall impressions 
than respondents, participants were divided into three groups: 1) those who responded to no rater 
perspective questions (Group 1), those who responded to at least one question (Group 2), and those 
who responded to all four questions (Group 3). ANOVAs were then performed to compare the three 
groups on the two overall impression items. The ANOVAs were significant for both Item 29, F(2,24080) = 
1033.53, p < .001, Np = .079; and Item 30, F(2,23653) = 1078.50, p < .001, Np = .084. For both items, the 
pattern of mean ratings was Group 3 > Group 2 > Group 1 (see Table 37). This suggests that individuals 
who elected not to respond to questions about their perspectives tended to have lower overall 
impressions of the administrator than did those who responded. Those who gave lower impressions of 
the administrator may have been reluctant to provide any information that might have led to their 
identity. Alternatively, those who revealed their perspectives may have been inclined to respond more 
positively than if they knew their responses would be completely anonymous. 


Table 37 


Means, Standard Deviations, and Post-Hoc Test Results for Personnel Ratings of Overall Impressions by 


the Number of Rater Perspective Questions Completed 


Number of Demographic Items Personnel Completed 


. No Items Completed 
Overall Impression 


Responded to at Least 1 
But Not All Items 


Responded to All Items 


M SD N M SD N M SD N tb 
op Wire! Rind.oF Job (stile 2.91, 1.08 1,246 | 3.13, 1.12 5,934 | 3.76, 1.02 16,903 | .079 
administrator doing? 

30;Does this administrator have. 599. -go7- ates | S6a¢. “406° S70%:| S's6.- ORO - 46.761 |*..084 


your confidence? 


Note. 


Means and standard deviations were computed from disaggregated database. 


Means with different subscripts within rows significantly differ (Tukey HSD, @=.01). 


SD = Standard deviation. 


Personnel responded on a 1 (Poor) to 5 (Superb) scale for Item 29. 


Personnel responded on a 1 (Definitely Note) to 4 (Definitely Yes) scale for Item 30. 


Additional Investigation of Rater Perspective Items: 


Sections Il and Ill of the initial IDEA Feedback for Administrators Report showed “atypical 
subgroups” whose average ratings are at least .5 higher or lower than that for all respondents. To better 
understand the frequency of atypical subgroups, an analysis was conducted to determine (1) how 
frequently each demographic subgroup in Items 34 to 37 had n greater than or equal to 5 respondents, 
and (2) of those, how many times the subgroup mean differed at least |.5| from the grand mean for a 
particular administrator. This was done for Items 1 to 30. Table 38 summarizes the results of those 
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analyses. The unweighted mean proportion of times a difference of |.5| occurred across all subgroups 
was .20. Clearly, some subgroup means differed by a magnitude of |.5| more frequently than did others. 


Table 38 
Frequency and Proportion of Times Rater Perspective Subgroups Differed from Administrator’s Mean by 
Magnitude of |.5|/ 


Number of times Mean number of F F 
Proportion of time 


Item/Subgroup Sormorecasesin times differences differences s('5| 
subgroup >|.5| 

Item 34-Principle affiliation with 

institution 

1.Faculty 361 38.13 1 

2.Student 39 7.7 .20 

3.Administrator’s staff 311 50.03 16 

4.Meaningful contacts but don’t report 346 54.67 16 


Item 35-Principle type of contact with 
administrator 


1.Work on mutual responsibilities 299 60.07 .20 
2.Work under his/her supervision 422 65.23 15 
3. Administrator provides services to me 283 50.67 18 
4. Primarily informal interactions 254 53.7 21 


Item 36-Frequency of meaningful 
contact with administrator 


1.Daily 149 30.63 21 
2.At least once a week, but not daily 384 79.77 21 
3.One to three times a month 358 60.8 17 
4.One to three times a term 287 53.97 19 
5.Less than once aterm 181 52.93 .29 


Item 37-Length of time at institution 


1.First year 73 28.23 39 

2.One to two years 133 42.6 32 

3.Three to five years 274 63.4 23 

4. More than five years 598 33 .06 
Summary: 


Administrators received the highest ratings on administrative roles, interpersonal 
characteristics, and confidence when, at the time of appointment, they were expected to maintain the 
status quo rather than to make major changes. The same was true when, at the time of appointment, 
administrators perceived the unit’s reputation in terms of effectiveness to be positive or neutral. These 
extraneous factors should, therefore, be considered when interpreting findings from the Administrator 
Report. Consequently, items associated with those factors remain on the AIF-R. Other characteristics of 
the administrator—years at the institution, years in the position, years in the position at any institution, 
number of professionals in the unit, and number of support staff—were only negligibly related to 
administrative roles, leadership style, and interpersonal characteristics. These very low correlations 
offer evidence of consistency in interpretations across several administrator demographics. 
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Some elements of the rater’s perspective were related to ratings of administrative roles, 
personal characteristics, and overall summary judgments. The more frequently raters had contact with 
their administrator, the more likely they were to rate her or him positively on a number of 
characteristics. Many factors could have contributed to these relationships. Perhaps raters having more 
frequent contact were more affected by social desirability and therefore gave higher ratings. 
Alternatively, those in frequent contact may have experienced a more balanced view of their 
administrator than would those having minimal contact. Administrators might be more likely to 
establish meaningful relationships with the people they interact with regularly. Consequently, those in 
frequent contact might have been less affected by chance negative encounters, rumors, and innuendos. 


Other elements of the rater’s perspective that were related to ratings of the administrator 
included principal affiliation with the institution, type of contact with the administrator, and length of 
time at the institution. Given the number of variables associated with elements of the rater’s 
perspective, and the concern with social desirability, only one of four items was retained in the IA-R: 
principal affiliation with the institution. 


Although we have offered substantial evidence to support the validity of the items and 
subscales in the IA, we recognize that changes in item wordings in the IA-R could affect the relationships 
reported in this section. Given the consistency in inter-item relationships, we consider it unlikely, 
however, that the validity of the instruments would be greatly compromised by the slight modifications 
made in item wordings (mostly changes in verb form). However, the ultimate test of an evaluation 
instrument’s validity is in its use. To make valid use of the IDEA Feedback for Administrators system 
more achievable, the user is encouraged to carefully read the instructions on survey administration and 
The IDEA Feedback for Administrators Report-Revised. 


RELIABILITY AND STANDARD ERRORS OF MEASUREMENT 


Reliability refers to consistency in measurements. Of particular concern to users is the question 
of how consistently personnel within the same unit rate the administrator’s performance. Also, does this 
consistency vary, depending on the item or subscale? Does it vary by the number of raters in the unit? 
Errors of measurement remind users that all scores contain some amount of “wobble,” unexplained 
variance, or error. Any interpretations of scores provided in the IDEA Feedback for Administrators 
Report-Revised should therefore consider the imperfect nature of any measure. 


Estimates of Item Split-Half Reliabilities and Standard Errors of Measurement on Ratings of 
Administrative Roles 


Evidence of reliability can come from relationships among items within the same instrument 
administered on a single occasion (i.e., internal consistency coefficients). One estimate of internal 
consistency is found in the split-half method, which, in this case, correlates the scores of subgroups of 
raters. We began by selecting administrators for whom the number of raters equaled 12 or more. For 
administrators with more than 12 raters, 12 were randomly selected. We then performed 1,000 random 
splits (n = 6 each) on each administrator’s sample of raters and assigned splits to Groups A or B. We 
computed means on each item for Groups A and B by each split (1,000 means on each item for each 
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group). We then correlated the means of Group A and B for each item and computed the mean r for 
each item. The Spearman-Brown prophecy formula‘ was then used to estimate the reliability of the 
average respondent ratings of administrators with 8, 12, 16, and 20 raters. Standard errors of 
measurement (SEM) were then calculated’. Results are presented in Table 39. 


The reliability (r) coefficients in Table 39 are moderately high, especially when the number of 
raters is at least 16. Aggregating the individual ratings into Groups A and B restricted the variability in 
the ratings, which affected the size of the coefficients.° The limited variability also affected the standard 
errors of measurement.’ However, the standard errors of measurement indicate that, in most instances, 
the average personnel ratings were dependable. 


Table 39 
Estimates of Split-Half Reliabilities and Standard Errors of Measurement for Personnel Ratings of 
Administrative Roles (N = 732) 


Number of Raters 


8 12 16 20 
Administrative Roles r SEM r SEM r SEM r SEM 
1. Communicates and displays a visionary plan for the 58 36 65 33 etl 30 75 28 
future. 
2. Establishes sound priorities. 56 .36 .64 33 .69 30 .73 .28 
3. see knowledge/expertise required for this 62 31 70 28 75 25 728 24 


4. Makes wise decisions, judgments, and 
recommendations. 
5. Initiates actions that anticipate problems or resolves 


.62 35 .69 31 74 29 78 27 


52 .40 .60 37 .65 34 .69 32 


them. 
6. Is an effective "team" member. .62 39 70 35 75 32 78 .29 
7. Contributes positively to this institution’s image and 63 31 71 28 75 95 79 33 
reputation. 


8. Communicates relevant information to appropriate 
constituencies. 

9. Seeks opinions of others before establishing policies 
or procedures. 

10. Earns the trust and respect of those who come in 
contact with him/her. 


57 37 65 33 70 31 74 29 


.60 Al 67 37 72 34 76 32 


.65 38 72 33 77 30 .80 .28 


Note: SEM = Standard error of measurement. 


Estimates of Item Split-Half Reliabilities and Standard Errors of Measurement on Ratings of 
Personal Characteristics 


4 . Aare ; ; F ea : 
Tnx =__NX I, where r, is the reliability of a rating employing x raters and r,,is the reliability of a rating that has nx raters. For example, 
1+(n-1)r, — if computing the reliability of ratings when personnel averaged 16, n = 2, or twice those averaging 8 personnel. 


> sd N(-re] 

Correlation coefficients are substantially affected by the variability in the item responses being correlated. 
i The standard error of measurement provides a meaningful way of interpreting the reliability of each item. An administrator’s “true score,” or 
average, on an item could theoretically be obtained if an item were administered to the same group of personnel an infinite number of times. 
The standard error of measurement tells us the chances are about 2 in 3 that an administrator’s true score average rating on an item falls 
within the range of + 1 standard error around the obtained average rating. 
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Respondents used a 7-point semantic differential scale to rate their administrator on 18 
personal characteristics (Items 11 to 28). The direction of the scale varied: For half the items a “1” was 
associated with a positive characteristic; for the other half a “1” was associated with a negative 
characteristic. Prior to computing reliabilities, we therefore re-coded the data so that higher values for 
each personal characteristic represented more positive evaluations of the administrator. Estimates of 
split-half reliabilities and standard errors of measurement for personnel ratings were then computed. 
The split-half reliability coefficients presented in Table 40 are moderate, which is to be expected given 
the limited variability in the ratings after individual ratings were aggregated and randomly assigned into 
Groups A and B. The standard errors of measurement indicate that, in most instances, the average 
personnel ratings were dependable when considering the scale ranges from 1 to 7. 


Table 40 
Estimates of Split-Half Reliabilities and Standard Errors of Measurement for Personnel Ratings of 
Personal Characteristics (N = 732) 


Number of Raters 


8 12 16 20 
Personal Characteristics r SEM r SEM r SEM r SEM 
11. Indecisive vs. Decisive .32 54 38 51 43 49 47 .48 
12. Disorganized vs. Organized .42 57 49 53 54 50 59 .48 
13. Remote vs. Approachable 53 59 61 54 .66 50 .70 47 
14. Untruthful vs. Honest 39 54 46 51 51 48 55 .46 
15. Unfair vs. Fair .52 53 .60 .49 .65 45 .69 .43 
16. Autocratic vs. Democratic 57 .65 .65 58 70 54 74 50 
17. Unfeeling vs. Caring 45 54 53 .50 58 47 .63 45 
18. Manipulative vs. Straightforward 51 57 59 52 .64 49 .68 .46 
19. Inconsistent vs. Consistent 38 58 44 .55 50 52 54 .50 
20. Lethargic vs. Vigorous 53 45 .60 41 .66 38 70 36 
21. Ambiguous vs. Clear 45 55 53 51 59 48 .63 .46 
22. Self-centered vs. Institution-centered 54 52 61 .48 .67 44 71 41 
23. Insensitive vs. Understanding 52 52 .60 47 .65 44 .69 41 
24. Opinionated vs. Receptive to ideas 55 57 .63 52 .68 48 72 45 
25. Untrustworthy vs. Trustworthy .49 53 57 .49 .62 46 .67 .43 
26. Passive vs. Active .49 .50 57 .46 .63 43 .67 .40 
27. Aloof vs. Warm .60 54 .68 .49 73 5 77 41 
28. Predictable vs. Erratic .37 53 44 50 49 48 54 .46 


Note: SEM = Standard error of measurement. 


Estimates of Item Split-Half Reliabilities and Standard Errors of Measurement on Overall 
Summary Judgments 


The reliability coefficients in Table 41 show moderately high consistency among raters for both 
overall summary judgment measures, especially when the number of total raters exceeds 15. The r- 
values were somewhat higher for Item 29, which would be expected given the greater variability in that 
item’s five-point scale compared to Item 30’s four-point scale. The small standard errors evidence the 
dependability of both measures. 
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Table 41 
Estimates of Split-Half Reliabilities and Standard Errors of Measurement for Overall Summary Judgments 
(N = 732) 


Number of Raters 


8 12 16 20 
Summary Judgments r SEM r SEM r SEM r SEM 
29. What kind of job is this administrator 
doing? .67 34 74 30 .78 .27 .82 25 


30. Does this administrator have your 
confidence? 
Note: SEM = Standard error of measurement. 


61 .27 .69 .24 74 22 77 .20 


Internal Consistency of Scales 


The Cronbach alpha (a) coefficient provides another estimate of internal consistency by 
computing the average inter-correlation among items. High values are desirable. One would expect, for 
example, that items loading on the same factor should be highly inter-correlated. Although the 
Administrators Report reports neither factor composite scores nor scale scores, we provide internal 
consistency coefficients here to offer evidence of the internal consistency of the various dimensions of 
the IA-R. 


We began by computing Cronbach @ and item-total statistics on the 10-item scale for 
Administrative roles. The internal consistency of the total scale was high, Cronbach’s a = .98, SEM = .08. 
Table 42 presents item-total statistics. 


Table 42 


Internal Consistency Coefficients (Cronbach a) and Item-Total Statistics for Administrator Role Scale (N = 
732) 


Item-Total Statistics 


Items 
Item-total r a. if item deleted 
1. Communicates and displays a visionary plan for the future. 85 .98 
2. Establishes sound priorities. 93 .98 
3. Displays knowledge/expertise required for this position. .88 .98 
4. Makes wise decisions, judgments, and recommendations. .96 .98 
5. Initiates actions that anticipate problems or resolves them 93 98 
before they become major concerns. : : 

6. Is an effective "team" member. 94 .98 
7. Contributes positively to this institution’s image and 90 98 
reputation. : : 

8. Communicates relevant information to appropriate 92 98 
constituencies. ; ; 

9. Seeks opinions of others before establishing policies or 

89 .98 
procedures that affect them. 
10. Earns the trust and respect of those who come in contact 95 98 


with him/her. 
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Next, we computed internal consistency coefficients for the two scales that comprised the 
administrator’s personal characteristics. Both Personal Characteristics ((a = .96, SEM = .15) and 
Leadership Style (a = .91, SEM = .18) had high internal consistency. Table 43 contains item-total 
statistics for these scales. 


Table 43 
Internal Consistency Coefficients (Cronbach co) and Item-Total Statistics for Interpersonal Characteristics 
and Leadership Style Scales (N = 732) 


Item-Total Statistics 


Items 

Item-total r a if item deleted 
Personal Characteristics 
16. Autocratic vs. Democratic .88 96 
23. Insensitive vs. Understanding 92 .96 
24. Opinionated vs. Receptive to ideas .88 .96 
15. Unfair vs. Fair 91 .96 
13. Remote vs. Approachable .80 .96 
18. Manipulative vs. Straightforward .87 .96 
25. Untrustworthy vs. Trustworthy .86 .96 
22. Self-centered vs. Institution-centered 81 96 
Leadership Style 
11. Indecisive vs. Decisive 82 88 
12. Disorganized vs. Organized .76 89 
26. Passive vs. Active .66 91 
21. Ambiguous vs. Clear .83 .88 
19. Inconsistent vs. Consistent 79 88 


Estimates of Item Stability 


Stability—the consistency of a measure across time—is another important indicant of reliability. 
To examine the stability of the personnel ratings of the administrator, results for 125 administrators 
who participated in the IDEA Feedback for Administrators system on at least two different occasions 
were analyzed. If an administrator participated more than twice, the participation closest in time to that 
of the first occasion was used. The time between ratings were submitted varied by administrator. Table 
44 shows the Pearson r “test-retest” correlations for all items on the IA. For items retained in the IA-R 
(IA-R item numbers indicated in parentheses), coefficients ranged from .58 to .75, with a median r= .67. 
This indicates that moderately strong stability existed between performance ratings of the same 
administrators. One would not expect extremely high test-retest correlations because changes should 
occur in behaviors and ratings. Some administrators most likely made more improvements than others. 
They might, therefore, have changed their ordinal positions across time, which would have reduced the 
magnitudes of the correlations. 
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Table 44 
Pearson r Correlations between Personnel Ratings of the Same Administrator on Two Different Occasions 
(N = 125) 


IA Item (IA-R item number) Pearson r 
1. Communicates and displays a visionary plan for the future. (1) .70 
2. Establishes sound priorities. (2) 61 
3. Displays knowledge/expertise required for this position. (3) 58 
4. Makes wide decisions, judgments, and recommendations. (4) 59 
5. Initiates actions that anticipate problems or resolves them. (5) .60 
6. Is an effective "team" member. (6) .67 
7. Contributes positively to this institution’s image and reputation. (7) 70 
8. Communicates relevant information to appropriate constituencies. (8) .66 
9. Seeks opinions of others before establishing policies or procedures. (9) 71 
10. Earns the trust and respect of those who come in contact with him/her. (10) .70 
11. Indecisive vs. Decisive (11) .66 
12. Disorganized vs. Organized (12) 73 
13. Remote vs. Approachable (13) 63 
14. Untruthful vs. Honest .66 
15. Unfair vs. Fair (14) 63 
16. Autocratic vs. Democratic (15) .67 
17. Unfeeling vs. Caring 73 
18. Manipulative vs. Straightforward (16) 71 
19. Inconsistent vs. Consistent (17) .64 
20. Lethargic vs. Vigorous .62 
21. Ambiguous vs. Clear (18) 57 
22. Self-centered vs. Institution-centered (19) .68 
23. Insensitive vs. Understanding (20) 75 
24. Opinionated vs. Receptive to ideas (21) 74 
25. Untrustworthy vs. Trustworthy (22) 72 
26. Passive vs. Active (23) 65 
27. Aloof vs. Warm 76 
28. Erratic vs. Predictable 64 
29. What kind of a job is this administrator doing? .69 
30. Confidence in administrator 53 
Summary 


Three sources of evidence were provided regarding the reliability of personnel ratings of the 
administrator. First, the split-half method showed that responses to individual IA items have acceptable 
reliability, especially when the number of raters exceeds 15. Similarly, standard errors of measurement 
supported the dependability of individual items, especially as the number of raters increases. Second, 
Cronbach a@ coefficients were high for the Administrative roles, Interpersonal Characteristics, and 
Leadership Style scales. Third, test-retest correlation coefficients revealed that personnel ratings had 
good stability across time. 
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Multiple sources of evidence have been presented to support the reliability of personnel 
responses to items retained in the IA-R. Ultimately, though, the user must determine whether local 
scores are sufficiently trustworthy to warrant use and interpretation. The number of raters and the care 
administrators and personnel take in completing the ratings are important considerations. 
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Appendix A: Administrator Information Form (AIF) and Impressions of 
Administrator (IA) 
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IDEA Feedback for Administrators 


Administrator Name: Sample Administrator 


Administrator's Title 
_IDEA University 


VERIFY? 


IMPRESSIONS OF ADMINISTRATORS 


Results from this survey will be used both to assist in appraising professional functioning and to suggest 
how that functioning could be improved. Therefore, it will be helpful if you can identify both strengths 
and weaknesses. 


To encourage your honesty and objectivity, you are asked not to identify yourself. Results of the survey 
will be summarized for all respondents and for various groups, but not for individuals. Individual identity 
will be protected as group analyses will be reported only for groups where there are at least five 
respondents. However, your responses to open-ended questions will be provided verbatim to the 
administrator. Thus, you will want to avoid comments that would reveal your identity. 


Strengths and Weaknesses 


To what degree do each of the following constitute a “strength” or “weakness” of the administrator? 
Mark the circle which best corresponds to your judgment, using the following scale: 


= Definite Weakness 
= More a Weakness than a Strength 
= In Between 
= More a Strength than a Weakness 
= Definite Strength 

J = Cannot Judge 


QU RWNPR 


© © © © © oO Communicates and displays a visionary plan for the future. 
© © © 0 0 o- Establishes sound priorities. 
Displays knowledge/expertise required for this position. 


© © © © 0 oO Makes wise decisions, judgments, and recommendations. 


oP WN Pp 
e) 
oO 
oO 
e) 
Oo 
Oo 


© © © 0 0 0 Initiates actions that anticipate problems or resolves them before they become major 
concerns. 


6 co © 0 © o o Isaneffective “team” member. 

7. © © © © © © Contributes positively to the institution’s image and reputation. 

8.0 © © © © o Communicates relevant information to appropriate constituencies. 

9. 0 © © © © oo Seeks opinions of others before establishing policies or procedures that affect them. 


10.° © © © © o- Earns the trust and respect of those who come in contact with him/her. 


The Administrator’s Personal Characteristics 


In this section, please describe the administrator’s personal attributes by marking the appropriate number 
on each scale. Mark “Cannot Judge” (CJ) if you have little or no basis for making a rating. 


123 4 5 6 7 CJ 
11.0 © 0 © © 0 o o Decisive (1) vs. Indecisive (7) 
12.0 0 0 © 0 0 © o- Organized (1) vs. Disorganized (7) 
13.0 0 © © 0 © © o- Remote (1) vs. Approachable (7) 
14.0 0 © © 0 © © o- Honest (1) vs. Untruthful (7) 
15.0 0 © © 0 0 o o- Unfair (1) vs. Fair (7) 
16.0 0 © © 0 0 o o Autocratic (1) vs. Democratic (7) 
17.0 0 0 © 0 © © oOo Caring (1) vs. Unfeeling (7) 
18.0 © © © © © © oO Manipulative (1) vs. Straightforward (7) 
19.0 0 © © 0 © o o- Consistent (1) vs. Inconsistent (7) 
20.0 © © 0 © 0 o o  Lethargic (1) vs. Vigorous (7) 
21.0 © © © © 0 © o Ambiguous (1) vs. Clear (7) 
22.0 0 0 © 0 0 o o Self-centered (1) vs. Institution-centered (7) 
23.0 © © © © © 0 o Understanding (1) vs. Insensitive (7) 
24.0 0 0 © © © 0 oo Receptive to ideas (1) vs. Opinionated (7) 
25.0 © 0 0 © 0 oo Trustworthy (1) vs. Untrustworthy (7) 
26.0 0 0 © © 0 0 oo Passive (1) vs. Active (7) 
27.9 © © 0 © 0 o o Aloof (1) vs. Warm (7) 
28.0 0 0 0 © 0 o o Predictable (1) vs. Erratic (7) 


Overall Impressions 


29. What kind of a job is this administrator doing? 
o Poor 

o Mediocre 

o Good 

o Excellent 

o Superb 

o Can't judge 


30. Does this administrator have your confidence? 
o Definitely not 

No, but | have reservations about this. 

Yes, but | have reservations about this. 

Definitely yes. 

| have formed no stable opinion. 


oO 0 0 


Open-ended Comments 


31. What are this administrator’s main assets? 


32. What reservations do you have about this person as an administrator? 


33. What changes (e.g., in priorities, style, organization, policy) would do most to improve this administrator’s 
effectiveness? 


Rater Perspective 


34. What is the principal way that you are affiliated with this institution? 


oO O00 0 0 


| am a faculty member. 

| am a student. 

| am a member of this administrator’s staff. 

| have meaningful contacts with this administrator but don’t report to this person. 
| choose to omit my response to this question. 


35. What is the principal type of contact you have with this administrator? 


oO :0 0 0-0 


We work together on mutual responsibilities or assignments. 
My work requires his/her supervision, approval, or judgment. 
This administrator provides services that | want or need. 
Our contact consists primarily of informal interaction. 

| choose to omit my response to this question. 


36. How often do you have meaningful contact with this administrator? 


oO0O00 0 0 


On a daily basis 

At least once a week, but not daily 

One to three times a month 

One to three times a term 

Less than once a term 

| choose to omit my response to this question 


37. How long have you been at this institution? 


Oo O00 0 0 


This is my first year 

One to two years 

Three to five years 

More than five years 

| choose to omit my response to this question 


If you Submit you will no longer be able to access the survey or modify your responses. 
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Adiministrator Infonnation Form 


Administrator Name: Sample Administrator 


Administrator's Title 
_IDEA University 


VERIFY? 


ADMINISTRATOR INFORMATION FORM (AIF) 


Please complete the following survey to the best of your knowledge. The information submitted on this 
form is used for research purposes only. Your answers will remain confidential and do not affect the 
results. 


General Information 


1. Number of years at institution 
2. Number of years in this position at this institution 
3. Number of years in this position at any institution 


Please mark only one response for the following questions. 


4. When you first assumed this position, what was the general campus reputation of your unit in terms of 
importance/influence? 
Oo positive 
O neutral 
Oo negative 
Oo NA/new unit 
oO don’t know 
5. When you first assumed this position, what was the general campus reputation of your unit in terms of 
effectiveness? 
O positive 
O neutral 
O negative 
Oo NA/new unit 
oO don’t know 
6. When you first assumed this position, you were expected to: 


O make major changes in the unit’s activities and/or the way they were conducted 
O primarily maintain established services and procedures 
oO other 


7. Before you assumed this position, turnover of personnel in the unit had been: 
O higher than most units 

about average 

lower than most units 

don’t know 

I'd prefer not to respond 


oOoO0°0 


8. Since you have been in this position, personnel turnover has: 
© increased 


O remained constant 
O decreased 
oO don’t know 
oO I'd prefer not to respond 
9. Since you have been in this position, financial support for the unit, relative to that for other units at the 


institution, has: 
© increased 


O remained constant 
O decreased 
oO don’t know 
oO I'd prefer not to respond 
10. Since you have been in this position, the workload of the unit has: 
Oo increased 
O remained constant 
O decreased 
oO don’t know 
oO I'd prefer not to respond 
11. Including interim leadership, approximately how many individuals, including yourself, have occupied your 


position during the past ten years? 
12. Number of people in this unit (omitting students) classified as professionals? 


13. Number of people in this unit (omitting students) classified as support staff? 


If you Submit you will no longer be able to access the survey or modify your responses. 


Copyright © 2004, IDEA Center 


Appendix B: Revised Administrator Information Form (AIF-R) and Impressions of 
Administrator (IA-R) 
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IDEA Feedback for Adiministrators 


IDEA Administrator Name: Sample Administrator 
Administrator's Title 


ONLINE _IDEA University 


VERIFY? 


IMPRESSIONS OF ADMINISTRATORS 


Results from this survey will be used to assist in appraising professional performance and to suggest how 
performance could be improved. Either purpose will be best served if you reflect on the administrator’s overall 
performance and provide a fair and honest representation of the administrator. Therefore, your responses 
should identify both strengths and weaknesses. 


To encourage your honesty and objectivity, you are asked not to identify yourself. Results of the survey will 
be summarized for all respondents, but not for individuals. Do realize, however, that your responses to open- 
ended questions will be provided verbatim to the administrator. Thus, you should avoid comments that could 
reveal your identity. 


Completing this survey takes approximately 20 minutes, so please allow sufficient time to provide thoughtful, 
fair, and honest responses. 


l. The Administrator’s Roles 


The list below describes roles that administrators perform. Please rate the administrator on each of the roles 
by selecting the number that best represents your judgment of how well he/she performed over the past year 
as an administrator. Select the number which best corresponds to your judgment, using the scale below. 
Select “Cannot Judge” (CJ) if you have little or no basis for making a rating. 


= Definite Weakness 
= More a Weakness than a Strength 
= In Between 
= More a Strength than a Weakness 
= Definite Strength 

J = Cannot Judge 


QOURWNR 


o © © © © oO Communicating a visionary plan for the future 
o © © oo o- Establishing sound priorities 
o 0 0 © © oO Displaying knowledge/expertise required for this position 


o © © © 0 oOo Making wise decisions, judgments, and recommendations 


© 0 © © 0 oo Being an effective team member 
-O© © © © © © Contributing positively to the institution’s image 


o © O© © © © Communicating relevant information to appropriate constituencies 


poe OeN Pat YN PF 
fo) 
° 
fo) 
fo) 
° 
° 


o © © © 0 oo Seeking opinions of others before establishing policies or procedures that affect them 


= 
oO 
(e) 
fe) 
fe) 
10) 
fe) 
(e) 


Earning the trust and respect of others 


Initiating actions that anticipate problems or resolving them before they become major concerns 


Part Il. The Administrator’s Personal Characteristics. 


This section includes 15 pairs of adjectives describing the administrator’s personal characteristics that have 
been recognized as important to performance. One adjective is anchored with a “1” and the other with a “7.” For 
each pair, select the number that best describes the administrator along the continuum. If each of the adjectives 
is equally descriptive, select the middle score (4). Select “Cannot Judge” (CJ) if you have little or no basis for 
making a decision. 


12 3 4 5 6 7 CJ 
11. Indecisive 0 0 0 0 0 0 0. Decisive 0 
12. Disorganized O 0 0 0O 0 0 O- Organized O 
13. Remote 0 0 0 0 0 0 0 Approachable O 
14. Unfair 0 0 0 0 0 0 0 Fair 0 
15. Autocratic 0 0 0 0 0 0 oO Democratic 0 
16. Manipulative oO 0 0 0 0 Oo oO Straightforward O 
17. Inconsistent 0 0 0 0 0 0 O- Consistent 0) 
18. Ambiguous 0 0 0 0 0 0 0 Clear O 
19. Self-centered 0 0 0 0 0 OO Institution-centered o 
20. Insensitive 0 0 0 0 0 0 oO Understanding O 
21. Opinionated 0 0 0 0 0 0 0 Receptivetoideas oO 
22. Untrustworthy 0 O O 0 0 oO oO Trustworthy fe) 
23. Unimaginative 0 0 O OO 0 0 oO _ Innovative O 
24. Rigid 0 0 0 0 0 0 oO Flexible O 
25. Impractical 0 0 0 0 0 0 0 Practical O 


Part Ill. Overall Impressions. 
Please select your response to the following questions. 


26. Overall, this administrator has provided excellent leadership. 
oO Strongly Disagree 
oO Disagree 
Oo In Between 
oO Agree 
oO Strongly Agree 
Oo Cannot Judge 


27. | have confidence in the administrator's ability to provide future leadership in this position. 
Strongly Disagree 

Disagree 

In Between 

Agree 

Strongly Agree 

Cannot Judge 


(e) 


OoO0O000 


Part IV. Open-ended Comments. 


Responses to open-ended questions are provided verbatim to the administrator. You should, therefore, avoid 
comments that could reveal your identity. 


28. What are this administrator’s main strengths? 


29. How might this administrator improve his or her performance? 


30. What are the most important challenges facing this administrator? 


Part V. Rater Perspective [Optional Item] 


Please select only one response for the following question. Demographic subgroup results are reported to the 
administrator only if there are at least 5 individuals responding from a specific category. 


31. What is the principal way that you are affiliated with this institution? 


Faculty member. 

Student. 

Administrator’s staff. 

Professional colleague/associate 

| choose to omit my response to this question. 


If you Submit you will no longer be able to access the survey or modify your responses. 


8-0. 0 'O=0 
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IDEA Feedback for Adiministrators 


Administrator Name: Sample Administrator 
Administrator's Title 


ONLINE _IDEA University 


VERIFY? 


ADMINISTRATOR INFORMATION FORM 
Part Il. Your Administrative Roles. 


The list below describes roles that administrators perform. Please rate yourself on each of the roles by selecting 
the number that best represents your judgment of how well you performed over the past year as an administrator. 
Select the number which best corresponds to your evaluation of yourself, using the following scale: 


1 = Definite Weakness 

2 = More a Weakness than a Strength 
3 = In Between 

4 = More a Strength than a Weakness 
5 = Definite Strength 


© © © oO oO Communicating a visionary plan for the future 

© © 0 oo Establishing sound priorities 

© oOo 0 © o_ Displaying knowledge/expertise required for this position 

© © © 0 o- Making wise decisions, judgments, and recommendations 

Initiating actions that anticipate problems or resolving them before they become major concerns 
o oOo 0 © © Being an effective team member 

© © © © © Contributing positively to the institution’s image 


© © © oO oO Communicating relevant information to appropriate constituencies 


fo oN Oa F ON BP 
° 
° 
° 
° 
° 


© © © 0 o- Seeking opinions of others before establishing policies or procedures that affect them 


= 
o 
[e) 
[e) 
[@) 
[e) 
[@) 


Earning the trust and respect of others 


Part Il. Your Personal Characteristics. 


The section includes 15 pairs of adjectives. One adjective is anchored with a “1” and the other with a “7.” For each 
pair, select the number that best describes you along the continuum. If each of the adjectives describes you equally, 
select the middle score (4). 


123 4 5 6 7 
11. Indecisive 0 0 0 0 0 0 0. Decisive 
12. Disorganized O O 0 0 0 0 O- Organized 
13. Remote 0 0 0 0 0 0 Oo Approachable 
14. Unfair 0 0 0 0 0 0 0 Fair 
15. Autocratic 0 0 0 0 0 0 oO Democratic 
16. Manipulative oO 0 0 0 0 oO 0O_ Straightforward 
17. Inconsistent 0 0 0 0 0 0 OO Consistent 
18. Ambiguous 0 0 0 0 0 0 Oo Clear 
19. Self-centered 0 © 0 O 0 OO _Institution-centered 
20. Insensitive 0 0 0 0 0 0 oO Understanding 
21. Opinionated 0 0 0 0 0 0 0 Receptive to ideas 
22. Untrustworthy 0 O O O 0 oO 0 Trustworthy 


23. Unimaginative 0 0 O 0O OO 0O oO _ Innovative 
24. Rigid 0 0 0 0.0 0 0 Flexible 


25. Impractical 0 0 0 0 0 0 0 Practical 


Part Ill. Your Position. 
Please select only one response for the following questions about your unit when your first assumed this position. 


26. When you first assumed this position, what was the general campus reputation of your unit in terms of 
importance/influence? 


O positive 

© neutral 

O negative 

O not applicable/new unit 
© don’t know 


27. When you first assumed this position, what was the general campus reputation of your unit in terms of organizational 
effectiveness? 


O positive 

© neutral 

O negative 

© not applicable/new unit 
© don’t know 


28. When you first assumed this position, you were expected to: 
O make major changes in the unit’s activities and/or the way they were conducted 
O primarily maintain established services and procedures 
O other 


Part IV. General Information. 


29. Including this year, number of years at this institution _. 

30. Including this year, number of years in this position at this institution 

31. Number of years in this position at any other institution 

32. Including interim leadership, approximately how many individuals, including yourself, have occupied your position during 
the past 10 years? 

33. Number of people in this unit (omitting students) who report directly or indirectly to you? 


34. Of those, how many people report directly to you? 
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Appendix C: Focus Group Themes 


Overall Comments 


The instrument is “objectifying the subjective”; More comprehensive picture than the student ratings; Like the 
balance; could be longer, but then wouldn’t use it; Shows where can improve if want to; Can show those that 
evaluate — provides some good information; Can show the boss that giving good input 

Neutral to positive of overall form; useful associations; relationship between results and work needs to be 
extrapolated; need to follow the cautions 

Because the survey comes from a third party, it is credible; use one instrument for administrators so it is 
consistent; reports are difficult to decipher; need something more narrative about what that means for you. 
Neutral to positive of overall system; thinking of collective faculty — some of what you do is confidential and 
faculty don’t see everything 

Value added; need consistent metric to build personal profile; chair form is by far superior; 

This is one piece that is useful, but need to bring in other pieces 

Need option to add additional questions 


Role of IDEA with Improvement Process 


National campaign for improvement of HE leader 

Links to materials — papers 

Use research to plan program to develop — administrator wannabe training 

Best practices 

Training to understand what report telling you 

Grow in sophistication of use 

Like Educause — Develop a process model for [IT] leaders 

Could users sign up for different levels of use? For example, Executive User — 3 visits, access to tools, ramp us 
services as institution needed, help identify strengths, Partner with Gallop University Worldwide conferences 
Longitudinal look — value added — are they benefitting from usage? Chart some things 

Training — how administrative team make use of IDEA over time 


Impressions of Administrator Survey 
General 


Strengths & Weaknesses 


Nebulous item — makes “wise” decisions. More measurable language/items. Consider “sound” instead of wise; 
or are decisions consistent with mission of the university 

Doesn’t talk about measureable outcomes — meeting strategic goals, data collection, data driven decisions 

“In Between — provided with no information (some disagreement about usefulness of anchor) 

Cannot Judge — add /No Opinion — not enough information to make a judgment 

Dept. Chair form is superior due to ability to define "weight" the importance of job elements 

Ability to include audience specific questions 

Ability to select from a set of pre-defined questions 


Personal Characteristics 
Which of these are key? Trustworthy 
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Overall Impressions 
e Job as administrator needs global, big picture item 


e 29/30- what is purpose? (29. What kind of job is administrator doing? 30. Does this administrator have your 
confidence?) 

e Response options to 29 are more slanted to the positive. Why don’t we use a 1=negative to 5=positive scale, 
because meaning of words change every year. Always baggage with how words are used. 

e Excellent is better than Superb. Option — Poor, Mediocre, Adequate, Good, Excellent, Can’t Judge 

e Ask about “effectiveness” instead of “job” (in a “profession” so sensitive to the word “job”). 

e 30is an important question 

e Discussion about using a numerical scale instead of categories for responses. But, sometimes numbers aren’t 
helpful and this seems like one of those areas. 

e “reservations” is passive and wordy 


Comments 
e Instead of assets use strengths 
e 32. Reservations — don’t like; aren’t they also what you’d like to see them improve? 
e 32 and 33 are redundant - like 33 better — more constructive. 
e Reverse order 33 then 31 
e Group comments by respondent so they know how many & what group of questions came from same person 


Rater Perspective 
e 34. Institution Affiliation: Instead of “| am a member of this administrator’s staff” try “I report to this 


administrator.” (issue — hard for chairs or deans to respond to this item if survey is for a Provost) 
e 37. Three “to” five years 
e The smaller the institution, the more descriptive of the person when in combination 


Information Form 
e Information Form not linked to survey — How important is this to the job? 
e Even though the instructions say “research purposes” it is not clear — they wondered why these results are not 
reported and how is it used. (Be more clear) 
e 3. “in the unit” — what does this mean? Some units need high turnover to develop staff (IT units). 
e Items 3, 4, 7 all relate to turnover — what is the relationship of turnover to quality? Stability? 
e What about questions related to creating a strategic plan? Implementing? What about an outcome assessment 
question? Example — Of your major (strategic) goals for the year, how many (what %) were met? 
e 5. —based on competitive approach; what if all units have increased (or decreased)? 
8. Does this include faculty? Administrators? Number of people completing the survey? 


Self-rating 
e = This is how | perceive myself — would be interesting but not useful. 


e Or-This is how | think “they” perceive me 
e Need to ask — would your superior find it useful? 


Report 


General Comments 
e Useful, but not easily perceptive — if dirt simple might not be useful 
e Need to get layout expert to help design (p. 2 font too big; p. 3 make easier) 
e Like the atypical subgroup — consider how to highlight outliers 
e Raw data report is missing, maybe provide distribution 
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Explore norming — harder to do, but might be useful on qualities 
Explore demographics — director/associate dean, etc. 

Chart — longitudinal 

Charting on individual reports with narrative describing 
Simplify, consider narrative description 

Don’t know how to interpret atypical groups. 

Dashboard where automatically marked 

Refer to material on our website 

Adjust instructions/form for small departments/units. Anonymity is an issue but find a way to do it so relevant 
and usable 

Report raw data with small groups, even if no subgroups 

Some tables difficult to see 

Online reporting increases ability to use color 


Section |. Overall effectiveness 


Section Il. Strengths and Weaknesses in Performing Administrative roles 


Loves this section — on personal report she was seen as “autocratic.” This is not how sees self, but comes 
through to some. Saw that chairs don’t know her. Was a trigger to change. 

Reflects perceptions — right or wrong — need to be aware in order to change. 

Could add pairings for respondents 

Access a list of characteristics from a database [ABG: or add space for respondents to add/list additional 
characteristics] 

This is the “zingable” section. 

As evaluator (supervisor) — confirming and helpful. When evaluate other people and may know why if someone 
does get “zinged” — but still see as honest. 

Some items are key 

Recommendation for interpretation were helpful to some (average 4.0 or percent “strength” exceeding 75); 
others want those statements substantiated — who says so, and what standard is being used? 

Subgroup codes — hard to link to actual item (e.g., client) because don’t have survey instrument to compare. 
“subordinate” is industrial language 


Section III. Administrative Style/Personal Characteristics 


Other comments: 


Develop and provide best practices 
Different levels of access and provide longitudinal component (history and temporal trends) 
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Appendix D: EASE 
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EVALUATION OF ADMINISTRATIVE STYLE AND EFFECTIVENESS 


KANSAS STATE UNIVERSITY 
Planning and Evaluation Services 


Administrator: Institution: Dat 


Both to improve personnel recommendations and to provide a basis for improving 
administrative practices, you are requested to answer the questions on this form as 
honestly and objectively as possible. You need not sign your name, since only group 
results will be compiled. Your participation will be most helpful if you identify both 
strengths and_weaknesses. 


L Rater Characteristics 
A. How is your work related to that of this administrator? 


__I report directly to this person. __] use the services provided by this person. 
___I report to someone who reports to this person. _I work as a colleague with this person. 
__I and this administrator report to the same person. __ Other 
B.How often do you have meaningful contact with this administrator? 
__On a daily basis. __One to three times a term. 
At least once a week, but not daily. __Less than once a term. 


___One to three times a month. 
C. How long have you been at this institution in your present department/unit? 


___This is my first year. _ Three to five years. 
___One or two years. __More than five years. 


Il. Attributes of the Administrator 


For the attributes listed below, circle the number which best describes the administrator. 
Use "1" or "2" if the first word or phrase is most descriptive; use "4" or "5" if the second 
word is most descriptive; use "3" if the administrator displays both attributes about equally. 
"CJ" indicates that you can’t judge this attribute. 


1. Friendly (1 or 2) vs. Unfriendly (4 or 5).............-- e223 e Aone Co: 
2. Organized (1 or 2) vs. Disorganized ( or 5)........... 12-23) 45) CJ 
3. Accepting (1 or 2) vs. Rejecting (4 or 5).............- D2 Sia oe Col 
4, Planful (1 or 2) vs. Undirected (4 or 5).............- 1 iS on Co 
5. Dedicated to institution (1 or 2) vs. Dedicated to self (4 or 5) iL Pa} 2b fy (04) 
6. Energetic (1 or 2) vs. Lethargic 4 or 5).........-.-.-- 12) 3i%45 (CJ 
a Blexibles Guo 2) ivsonugids 4 OD) sie oe euele «rere erences Le S4one Cl 
8. Sensitive to others (1 or 2) vs. Insensitive to others (4 or 5) LAZe Sa cole Cd 
9. Open to ideas (1 or 2) vs. Close-minded (4 or 5)......... iL YB) 4 ds OY 
10. Democratic (1 or 2) vs. Autocratic (4 or 5)............ 1.2) 3) 4.55) Co 
11. Structured (1 or 2) vs. Ambiguous (4 or 5)...........-. 123) 4a bs Ge 
12. Motivating (1 or 2) vs. Discouraging (4 or 5).........-. dee ted Oud 
1S pe Strone (lion 2) vse Weak=(4. or o)eetace ics: toes =) 1S 4 Ova ie) 
14. Realistic (1 or 2) vs. Unrealistic (4 or 5)............-- TP 2Sie4 cb) Cy 
15. Sophisticated (1 or 2) vs. Naive (4 or 5).......--+00e- 123745) BCI 


(OVER, PLEASE) 


ee_rlee cn tt I a, ts a aN a IE RICA OEE TO, HE 


I Strengths and Weaknesses 


To what degree do each of the following constitute a "strength" or a “weakness” of the 
administrator? Use the following scale to make your ratings: 


1 = Definite wealmess 4 = More a strength than a weakness 

2 = More a weakness than a strength 5 = Definite strength 

3 = In between CJ = Cannot judge 
16. Communicates and displays a visionary and futuristic plan... . LY 8 szb fj (04) 
17; Has developed sound prionities .... 2... os ees wee owe 1) 2Serae ow Co 
18. Displays knowledge/expertise related to work assignments... .. WE POANS) eh Ts) (04) 
TOGumMakes wiserdecisionSmaeiiis sekeicr sletehenea onic kes ersesueio es IN 23345) CS 
20. Initiates actions which resolve problems in a timely manner... 12.13) 74561 CS 
DinmOfrersySouNGadvice Nia tre Ne icnnere terse malciecel erect oben: pee eas hee Sits) a 63) 
22. Contributes positively to the institution’s image and reputation. LIQ 34h. Cy 
23. Informs me about matters which affect me and/or my work... . PV eo eed; 
24. Seeks opinions of others before suggesting policies or procedures 

Halen elaioan Hollen iolo.s  Gseld Grouao Hla Bkeeore olb ols CLO E52 S74 bl Cd: 


IV. Overall Ratings 
25. Overall, how effective do you think this administrator is? 


__Exceptional __Very good Good Satisfactory _ Needs improvement 


26. Does this administrator have your confidence? 


_Always _ Usually __Half the time Less than half the time 


V. Additionai Observations 


In this section, you are invited to make additional observations and suggestions for 
improvement. Wriiten responses will be typed before being sent to the administrator so that 
handwriting cannot be used to identify the rater. Use additional pages as needed. 


27. What are this administrator's main strengths? 


28. What reservations do you have about this person as an administrator? 


29. What changes (in priorities, style, organization, policy) do you feel would improve this 


administrator’s effectiveness? 


30. 


Other comments? 


nnn nnn ee UU aE a Un SSI nS nn ER 


a 


Thank you. Please return the completed form in the self-addressed envelope provided. 
A-2 : 
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IDEA Feedback for Administrators Report IDEA 
COE INGER 


Insight. Improvement. Impact. e 


William McKinley 
Assistant Dean 
IDEA University 
08-31-2012 


Table of Contents 
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Respondent Information 


Number asked to participate: 240 
Number responding: 230 
Response rate: 95 % 
Students 30 
Deans 15 
Other 25 
Faculty 150 
Omit 10 
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1.Report Overview 


This report summarizes results from feedback provided through the IDEA Feedback for Administrators survey. 
Insight into your strengths and possible areas for improvement can guide your leadership development. The 
value of the feedback will be greatly enhanced if you set aside specific time for careful review and reflection 
and, if possible, have follow-up conversations with a trusted colleague. 


The report begins with a quick snapshot of your overall effectiveness based on respondents' answers to 
two questions: whether you have provided excellent leadership in the past and whether respondents have 
confidence in your ability to provide excellent leadership in the future. 


The next section highlights your areas of strength related to administrative roles, leadership style, and personal 
characteristics. These should provide insight into perceptions of your overall effectiveness as well as provide a 
foundation for your professional development plan. 


The report then provides respondent feedback about your performance on ten administrative roles. The data 
are displayed to show averages as well as percentages of respondents who rated you high or low for each of 
the administrative roles. Also, your self-ratings are compared to the respondents’ average ratings for each of 
the administrative roles. 


Next, the report provides respondent feedback on your leadership style and personal characteristics. As 
previously, the data are displayed to show averages as well as percentages of respondents who rated you high 
or low for each of the leadership style and personal characteristic identifiers. The data also compare your self- 
rating to the respondents’ average ratings for each of the areas. 


The report concludes with more detailed arrays of the data. 


Summarized feedback for any customized questions that were added are provided in a separate report. Also 
provided separately are typed verbatim responses made to open-ended questions. 


Some Thoughts on Using This Report 


In each of the following sections, some guiding questions and suggestions are provided to facilitate your 
personal reflection. Focus on what you do well and on what is most effective. Build on those strengths as you 
identify areas to improve. 
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2.Overall Effectiveness 


Your overall effectiveness was assessed by asking respondents to respond to two statements: 

¢ Overall, this administrator has provided excellent leadership. 

« | have confidence in the administrator's ability to provide future leadership in this position. 
Most administrators want to be told they are doing well, are good leaders, and that their colleagues have 
confidence in their future leadership. But, as administrators, it is also likely that there will always be some who 
view your work negatively. While we encourage you not to make too much out of small differences or a small 
number of negative responses, the key is recognizing when the negative impressions are pervasive enough 
that it could make it difficult to accomplish your responsibilities. The snapshot below shows you at a glance the 


percent of all respondents that rated you positively and negatively on each statement. Respondent feedback 
for the entire group, and for appropriate subgroups, is then displayed. 


Performance to Date and Confidence in Future Leadership 
Overall, this administrator has provided excellent leadership. 
Percent providing positive and negative ratings Average Rating 
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| have confidence in the administrator's ability to provide future leadership in this position. 
Percent providing positive and negative ratings Average Rating 
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The following sections of this report provide more detail about the specific traits of your leadership style and 
personal characteristics that likely influenced the ratings that you received on these "Overall Effectiveness" 
items. 
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3.Reviewing Your Strengths 


Respondents provided feedback about their impressions of your performance of key administrative roles, 
leadership style, and personal characteristics. For administrative roles, specific areas of strength are 
suggested when at least 70% of respondents rated you positively (giving ratings of 4 or 5). For leadership style 
and personal characteristics, strengths are suggested when at least 60% rated you positively (providing ratings 
of 6 or 7). Following is a summary of areas where you were rated most positively. 


Ideally, one would want to be rated "Good" or "Outstanding" in all of the areas surveyed regarding your job 
performance, but in reality such ratings are seldom the case. If several items appear below, you should 

take pride in those areas where your colleagues rated you positively and have confidence in you as an 
administrator. If, however, no items, or just a few appear in this "Reviewing Your Strengths" section, you should 
consider how to improve in the areas in which you were not given high ratings as detailed in Sections 4 and 5 
that follow. 


Administrative Roles 
4. Making wise decisions, judgments, and recommendations (81% positive) 
10. Earning the trust and respect of others (76% positive) 
1. Communicating a visionary plan for the future (75% positive) 
2. Establishing sound priorities (75% positive) 
7. Contributing positively to the institution's image (73% positive) 
5. Initiating actions that anticipate problems or resolving them before they become major concerns (71% positive) 
8. Communicating relevant information to appropriate constituencies (71% positive) 


9. Seeking opinions of others before establishing policies or procedures that affect them (70% positive) 


Leadership Style 
23. Innovative (77% positive) 
11. Decisive (69% positive) 
12. Organized (61% positive) 


Personal Characteristics 
14. Fair (71% positive) 
22. Trustworthy (69% positive) 
16. Straightforward (61% positive) 


19. Institution-centered (61% positive) 
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4.Deeper Reflection: Administrative Roles 


Along with your own self-ratings, perceptions of your performance of 10 key administrative roles were provided 
by your respondents using the same 5-point scale you used when you rated yourself. The display on the 
following pages shows how your self-ratings compare to the average ratings of all respondents. And, if enough 
individuals were included, how your self-ratings compare to those for subgroups is presented. In addition, the 
percent rating you positively and negatively on each role is provided. 


As you review the results, be sure to consider the contexts of your day-to-day work life (the daily interactions, 
the short and long-term projects in which you and your colleagues are involved, and so on). The questions 
below will help guide your personal reflection. 


What percent of respondents rated you positively (gave you a 4 or 5) or negatively (gave you a 1 or 2) 
on each of the roles? 

Administrators need to set their own standards for what is considered to be acceptable positive or undesirable 
negative ratings. In general, if 70% or more of your respondents provided a positive rating, then that particular 
role can be considered a strength. While you build on your strengths (listed first on pages 5 and 6, indicated by 
the overall percent positive ratings), you might focus improvement efforts on those roles where fewer than half 
(50%) of the respondents rated your performance positively or where more than 30% rated your performance 
negatively. 


Are perceptions generally consistent among all subgroups or are some unique? 

Understanding if there is a group that views your performance differently can provide insight into how your 
work and interactions may be impacting individuals differently. If you find a higher percentage of negative 
responses from one subgroup on a particular role - or multiple roles - you might review situations and 
experiences with that particular subgroup to help identify specific causes for that low rating. 


It is also important to assess whether unique subgroup responses are due to a single recent event or represent 
an ongoing issue. For example, if you received a high percentage of negative ratings for the descriptor 
Anticipates Problems from a particular subgroup, you may recall a situation close to the time the survey was 
administered where you acknowledge failing to anticipate and address an issue. This obviously is less of a 
concern than negative ratings that occur due to more general behavioral patterns. 


Administrative roles that likely merit the most careful review are those where you both received a high 
percentage of negative ratings and rated yourself markedly higher than did the respondents. If you cannot 
identify specific situations or experiences that may have led to negative ratings or to a markedly inflated rating 
on your part, you might seek the counsel of a trusted colleague. 


How similar is your self-rating to the average of all respondents and to the averages of your 
subgroups? 

From a practical perspective, respondent ratings that were over one-half point (.5) lower than your self- 
rating (to the left of the shaded area), indicate those roles where your own sense of your strengths is inflated 
compared to what the respondents think. These may be areas to examine further to try to understand why 
these different perceptions exist. Ratings to the right of the shaded area indicate characteristics in which the 
average respondent rating was at least one-half point higher than your self-rating. In such cases, you may be 
underestimating your performance compared to what your respondents think. 


In some cases the difference in perception may not be of great concern. For example, if you gave yourself 

a rating of 5 on an administrative role and the average for respondents was 4.3, you are still being viewed 
positively by those who rated you. In contrast, if you gave yourself a self-rating of 4 and the average for 
respondents (or a subgroup of respondents) was 3.0, there is greater reason to explore why those differences 
in perception exist. 
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Administrative Roles 


4. Making wise decisions, judgments, and recommendations 
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5. Initiating actions that anticipate problems or resolving them before they become major concerns 
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3. Displaying knowledge/expertise required for this position 
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6. Being an effective team member 
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5.Deeper Reflection: Your Leadership Style and 
Personal Characteristics 


Respondents rated you on the same 15 descriptors and same 7-point scale on which you rated yourself when 
you filled out the Administrator Information Form (AIF). Seven of the items generally indicate a particular 
leadership style while eight generally speak to respondent assessments of your personal characteristics. 


This report always assigns a 7 to the typically more desirable end of the spectrum for each trait so that ratings 
of 6 or 7 are generally preferred over low ratings of 1 or 2. For example, you and the respondents rated your 
Decisiveness on a scale where "Indecisive" was a 1 and where "Decisive" was a 7. While high ratings (6 or 

7) are generally preferred to low ratings (1 or 2), some effective administrators develop unique styles that 
depart markedly from this expectation. Results in this section should be considered within the context of the 
effectiveness ratings reported in earlier sections of this report and within the context of your administrative role. 


The following displays show results for Leadership Style and Personal Characteristics, respectively. 


What percent of respondents rated you positively (gave you a 6 or 7) or negatively (gave you a 1 or 2) 
on each of the styles/characteristics? 

In general, if 60% or more of your respondents provided a positive rating, then that particular characteristic can 
be considered a strength. Build on your strengths (listed in order of highest percent with overall positive ratings 
on pages 8-10). If 20% or more of respondents overall, or as part of a sub-group, rated a role negatively, then 
that area might be a focus of improvement. 


Are perceptions generally consistent among all subgroups or are some unique? 

In some cases the difference in perception may not be of great concern. In other cases, if you find a higher 
percentage of negative responses from one subgroup, you might review situations and experiences with that 
particular subgroup to help identify specific causes for a low rating. You may need to consider some specific 
strategies to address unique perceptions. 


Areas in which you both received a high percentage of negative ratings and rated yourself markedly higher 
than did the respondents will warrant more careful reflection. If you cannot identify specific situations or 
experiences that may have led to negative ratings or to a markedly inflated rating on your part, you might seek 
the counsel of a trusted colleague. 


How similar is your self-rating to the average of all respondents and to the average of your subgroups? 
From a practical perspective, respondent ratings that were over three-quarters of a point (.75) lower than your 
self-rating (to the left of the shaded area), indicate those styles or characteristics about which your own sense 
of your strengths is inflated compared to what the respondents think. This may be an area to examine further 
to better understand why these different perceptions exist. Ratings to the right of the shaded area indicate 
characteristics in which the average respondent rating was higher than your self-rating. In such cases, you 
may be underestimating your styles or characteristics compared to how others perceive them. 
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Leadership Styles 


23. Unimaginative vs. Innovative 
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17. Inconsistent vs. Consistent 
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19. Self-centered vs. Institution-centered Average Rating 
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6.Response Details 


The data provided in this section of the report allow you to be even more precise in understanding your 
feedback. 
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26. Overall, this administrator has provided excellent leadership. 0 6 36 88 77 13 4.1 0.8 
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